Wave changes
This commit is contained in:
27
Uni/20241210012703-fyp.org
Normal file
27
Uni/20241210012703-fyp.org
Normal file
@@ -0,0 +1,27 @@
|
||||
:PROPERTIES:
|
||||
:ID: 7d199fbe-b0e7-48fd-b8a1-793044dbea01
|
||||
:END:
|
||||
#+title: fyp
|
||||
#+filetags: :index:uni:
|
||||
|
||||
In this file i will include things related to my final year *project*.
|
||||
|
||||
- The idea:
|
||||
*A web platform for users to take notes with an ai assistant*
|
||||
|
||||
- Migrating to cloud databse.
|
||||
I was originally intending to use AWS RDS with an e2c instance, but realised that the learning curve was too great, and that i needed a simpler solution.
|
||||
On the <2024-12-09 Mon> I decided to try this: exporting the current vm i had running and run in on oracle cloud.
|
||||
The results for this:
|
||||
|
||||
The UML for this project: [[id:4a8edaed-9ebd-402b-9c5f-7a0cb4399102][uml_fyp]]
|
||||
@startuml
|
||||
Alice -> Bob: Authentication Request
|
||||
Bob --> Alice: Authentication Response
|
||||
|
||||
Alice -> Bob: Another authentication Request
|
||||
Alice <-- Bob: Another authentication Response
|
||||
@enduml
|
||||
|
||||
|
||||
[[id:26b2ed9a-cb81-4c43-bc63-6b3c8ffa3bf1][fyp-report-planning]]
|
||||
12
Uni/20241210152650-uni_moc.org
Normal file
12
Uni/20241210152650-uni_moc.org
Normal file
@@ -0,0 +1,12 @@
|
||||
:PROPERTIES:
|
||||
:ID: 797d6e3e-98eb-4bc7-88b6-e096ef7306ad
|
||||
:END:
|
||||
#+title: uni_moc
|
||||
#+filetags: :uni:index:
|
||||
* Modules:
|
||||
- [[id:5443ed1c-bb7f-4eb4-9c96-d12648dd2291][TPIS]]
|
||||
- [[id:556d10d1-1c74-4d9f-a398-39cb3bd5d935][AFP]]
|
||||
- [[id:c69e4c4d-2fb4-4cf1-a835-a235cf6db8e9][ISE]]
|
||||
- [[id:3acffb66-bc1a-4661-904f-c5447b3c3488][Advanced Networking]]
|
||||
* Final Year Project
|
||||
- [[id:7d199fbe-b0e7-48fd-b8a1-793044dbea01][FYP]] Final Year Project
|
||||
9
Uni/20241210152713-tpis.org
Normal file
9
Uni/20241210152713-tpis.org
Normal file
@@ -0,0 +1,9 @@
|
||||
:PROPERTIES:
|
||||
:ID: 5443ed1c-bb7f-4eb4-9c96-d12648dd2291
|
||||
:END:
|
||||
#+title: tpis
|
||||
#+filetags: :uni:
|
||||
|
||||
Teaching Programming In School
|
||||
- Proposal: [[file:~/master-folder/misc/proposal.pdf][Proposal for the module]]
|
||||
- <2024-12-10 Tue> - Presentation
|
||||
708
Uni/20241210232054-uml_fyp.org
Normal file
708
Uni/20241210232054-uml_fyp.org
Normal file
@@ -0,0 +1,708 @@
|
||||
:PROPERTIES:
|
||||
:ID: 4a8edaed-9ebd-402b-9c5f-7a0cb4399102
|
||||
:END:
|
||||
#+title: uml_fyp
|
||||
#+filetags: :uni:
|
||||
|
||||
|
||||
* Class diagram
|
||||
** class 2:
|
||||
@startuml
|
||||
skinparam classAttributeIconSize 0
|
||||
skinparam packageStyle rectangle
|
||||
skinparam linetype ortho
|
||||
left to right direction
|
||||
|
||||
' Entity Layer
|
||||
rectangle "Entity Layer" as EntityLayer #E3F2FD {
|
||||
class User {
|
||||
+id: Long
|
||||
+roles: Set<String>
|
||||
+firstName: String
|
||||
+lastName: String
|
||||
+username: String
|
||||
+password: String
|
||||
+confirmPassword: String
|
||||
+email: String
|
||||
+phoneNumber: String
|
||||
+picture: byte[]
|
||||
}
|
||||
|
||||
class Workspace {
|
||||
+id: Long
|
||||
+name: String
|
||||
+description: String
|
||||
+fileCount: int
|
||||
+noteCount: int
|
||||
+lastAccessed: LocalDateTime
|
||||
+dateCreated: LocalDateTime
|
||||
}
|
||||
|
||||
class Library {
|
||||
+id: Long
|
||||
+name: String
|
||||
+description: String
|
||||
+dateCreated: LocalDateTime
|
||||
+lastAccessed: LocalDateTime
|
||||
}
|
||||
|
||||
class Page {
|
||||
+id: Long
|
||||
+name: String
|
||||
+workspaceId: Long
|
||||
+content: Text
|
||||
+height: int
|
||||
+width: int
|
||||
}
|
||||
|
||||
class FileMetadata {
|
||||
+id: Long
|
||||
+fileUrl: String
|
||||
+userId: String
|
||||
+description: String
|
||||
+fileData: byte[]
|
||||
+fileName: String
|
||||
+fileType: String
|
||||
}
|
||||
|
||||
class AiChat {
|
||||
+id: Long
|
||||
+name: String
|
||||
+workspaceId: Long
|
||||
+createdAt: LocalDateTime
|
||||
+messages: List<AIChatMessage>
|
||||
}
|
||||
}
|
||||
|
||||
' Repository Layer
|
||||
rectangle "Repository Layer" as RepositoryLayer #E8F5E9 {
|
||||
class AiChatRepository
|
||||
class FileMetadataRepository
|
||||
class LibraryRepository
|
||||
class PageRepository
|
||||
class UserRepository
|
||||
class WorkspaceRepository
|
||||
class RegisterRepository
|
||||
class AuthRepository
|
||||
}
|
||||
|
||||
' Service Layer
|
||||
rectangle "Service Layer" as ServiceLayer #FFF3E0 {
|
||||
class AiChatService
|
||||
class AuthService
|
||||
class FileMetadataService
|
||||
class LibraryService
|
||||
class PageService
|
||||
class RegisterService
|
||||
class UserService
|
||||
class WorkspaceService
|
||||
}
|
||||
|
||||
' Controller Layer
|
||||
rectangle "Controller Layer" as ControllerLayer #F3E5F5 {
|
||||
class AiChatController
|
||||
class AiChatWebSocketController
|
||||
class AuthController
|
||||
class FileMetadataController
|
||||
class LibraryController
|
||||
class PageController
|
||||
class RegisterController
|
||||
class UserController
|
||||
class WorkspaceController
|
||||
}
|
||||
|
||||
' External Python Microservice
|
||||
rectangle "External NLP Module (Python FastAPI)" as ExternalNLP #FCE4EC {
|
||||
class FastAPINLPServer {
|
||||
+/extract_text
|
||||
+/keywords
|
||||
+/tfidf_keywords
|
||||
+/summarize
|
||||
+/ner
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
rectangle "Frontend Core" as FrontendCore {
|
||||
class AboutComponent
|
||||
class AccessibilityComponent
|
||||
class ContactComponent
|
||||
class ForgotPasswordComponent
|
||||
class LoginComponent
|
||||
class LoginService
|
||||
class PrivacyComponent
|
||||
class RegisterComponent
|
||||
class RegisterService
|
||||
class SecurityComponent
|
||||
class TermsComponent
|
||||
|
||||
LoginComponent --> LoginService
|
||||
RegisterComponent --> RegisterService
|
||||
|
||||
LoginService ..> AuthController : REST
|
||||
RegisterService ..> RegisterController : REST
|
||||
}
|
||||
|
||||
rectangle "Frontend Entities" as FrontendEntities {
|
||||
class AiChatComponent
|
||||
class DashboardComponent
|
||||
class DefaultLayoutComponent
|
||||
class HomeComponent
|
||||
class LibraryComponent
|
||||
class ProfileComponent
|
||||
class WorkspacePageComponent
|
||||
class WorkspacesComponent
|
||||
|
||||
class AiChatServiceFE
|
||||
class DashboardServiceFE
|
||||
class LayoutServiceFE
|
||||
class HomeServiceFE
|
||||
class LibraryServiceFE
|
||||
class ProfileServiceFE
|
||||
class WorkspacePageServiceFE
|
||||
class WorkspacesServiceFE
|
||||
|
||||
AiChatComponent --> AiChatServiceFE
|
||||
DashboardComponent --> DashboardServiceFE
|
||||
DefaultLayoutComponent --> LayoutServiceFE
|
||||
DefaultLayoutComponent --> LoginComponent
|
||||
DefaultLayoutComponent --> RegisterComponent
|
||||
HomeComponent --> HomeServiceFE
|
||||
LibraryComponent --> LibraryServiceFE
|
||||
ProfileComponent --> ProfileServiceFE
|
||||
WorkspacePageComponent --> WorkspacePageServiceFE
|
||||
WorkspacesComponent --> WorkspacesServiceFE
|
||||
|
||||
ForgotPasswordComponent ..> UserController : REST
|
||||
AiChatServiceFE ..> AiChatController : REST
|
||||
AiChatServiceFE ..> AiChatWebSocketController : WebSocket
|
||||
|
||||
DashboardServiceFE ..> WorkspaceController : REST
|
||||
DashboardServiceFE ..> PageController : REST
|
||||
DashboardServiceFE ..> AiChatController : REST
|
||||
|
||||
LayoutServiceFE ..> UserController : REST
|
||||
|
||||
HomeServiceFE ..> AuthController : REST
|
||||
|
||||
LibraryServiceFE ..> LibraryController : REST
|
||||
LibraryServiceFE ..> FileMetadataController : REST
|
||||
|
||||
ProfileServiceFE ..> UserController : REST
|
||||
|
||||
WorkspacePageServiceFE ..> PageController : REST
|
||||
WorkspacePageServiceFE ..> FileMetadataController : REST
|
||||
WorkspacePageServiceFE ..> AiChatController : REST
|
||||
|
||||
WorkspacesServiceFE ..> WorkspaceController : REST
|
||||
AboutComponent --> ContactComponent
|
||||
AboutComponent --> TermsComponent
|
||||
AboutComponent --> PrivacyComponent
|
||||
}
|
||||
|
||||
class AngularClient
|
||||
AngularClient --> FrontendCore
|
||||
AngularClient --> FrontendEntities
|
||||
AngularClient ..> FastAPINLPServer : REST
|
||||
|
||||
|
||||
' Entity Relationships
|
||||
User "1" --> "many" Library
|
||||
User "1" --> "many" Workspace
|
||||
Library "1" --> "many" Workspace
|
||||
Library "1" --> "many" FileMetadata
|
||||
Workspace "1" --> "many" Page
|
||||
Workspace "1" --> "many" FileMetadata
|
||||
Workspace "1" --> "many" AiChat
|
||||
|
||||
' Controller → Service
|
||||
AiChatController --> AiChatService
|
||||
AiChatWebSocketController --> AiChatService
|
||||
AuthController --> AuthService
|
||||
FileMetadataController --> FileMetadataService
|
||||
LibraryController --> LibraryService
|
||||
PageController --> PageService
|
||||
RegisterController --> RegisterService
|
||||
UserController --> UserService
|
||||
WorkspaceController --> WorkspaceService
|
||||
|
||||
' Service → Repository
|
||||
AiChatService --> AiChatRepository
|
||||
AuthService --> AuthRepository
|
||||
FileMetadataService --> FileMetadataRepository
|
||||
LibraryService --> LibraryRepository
|
||||
PageService --> PageRepository
|
||||
RegisterService --> RegisterRepository
|
||||
UserService --> UserRepository
|
||||
WorkspaceService --> WorkspaceRepository
|
||||
|
||||
' Repository → Entity
|
||||
AiChatRepository --> AiChat
|
||||
FileMetadataRepository --> FileMetadata
|
||||
LibraryRepository --> Library
|
||||
PageRepository --> Page
|
||||
UserRepository --> User
|
||||
WorkspaceRepository --> Workspace
|
||||
RegisterRepository --> User
|
||||
AuthRepository --> User
|
||||
|
||||
@enduml
|
||||
|
||||
** authentication sequence diagram:
|
||||
@startuml
|
||||
title Full Authentication Flow - Registration to Secured Access
|
||||
|
||||
skinparam shadowing true
|
||||
skinparam packageStyle rectangle
|
||||
skinparam handwritten false
|
||||
skinparam linetype polyline
|
||||
skinparam ParticipantPadding 20
|
||||
skinparam maxMessageSize 150
|
||||
|
||||
actor User
|
||||
|
||||
== Registration Process ==
|
||||
|
||||
box "Client" #E3F2FD
|
||||
participant UserClient as "User"
|
||||
end box
|
||||
|
||||
box "Backend - Registration" #E8F5E9
|
||||
participant RegisterController
|
||||
participant UserService
|
||||
end box
|
||||
|
||||
UserClient -> RegisterController : POST /register\n(username, password, confirmPassword)
|
||||
RegisterController -> RegisterController : Check if passwords match
|
||||
alt Passwords do not match
|
||||
RegisterController --> UserClient : 400 Bad Request (Passwords mismatch)
|
||||
return
|
||||
end
|
||||
RegisterController -> RegisterController : Hash password (BCrypt)
|
||||
RegisterController -> RegisterController : Load default profile image
|
||||
RegisterController -> UserService : createUser(user)
|
||||
RegisterController --> UserClient : 200 OK (User Created)
|
||||
|
||||
== Authentication Process ==
|
||||
|
||||
box "Backend - Authentication" #FFF3E0
|
||||
participant AuthController
|
||||
participant AuthenticationManager
|
||||
participant JwtUtil
|
||||
end box
|
||||
|
||||
UserClient -> AuthController : POST /auth/login\n(username, password)
|
||||
AuthController -> AuthenticationManager : Authenticate credentials
|
||||
alt Valid credentials
|
||||
AuthController -> JwtUtil : generateToken(userDetails, userId)
|
||||
AuthController --> UserClient : 200 OK + JWT
|
||||
else Invalid credentials
|
||||
AuthController --> UserClient : 401 Unauthorized
|
||||
return
|
||||
end
|
||||
|
||||
== Accessing a Protected API ==
|
||||
|
||||
box "Security Filter Chain" #F3E5F5
|
||||
participant JwtRequestFilter
|
||||
participant UserDetailsService
|
||||
participant JwtUtilFilter as "JwtUtil"
|
||||
end box
|
||||
|
||||
UserClient -> JwtRequestFilter : Request with Authorization header
|
||||
JwtRequestFilter -> JwtUtilFilter : extractUsername(jwt)
|
||||
JwtRequestFilter -> UserDetailsService : loadUserByUsername
|
||||
JwtRequestFilter -> JwtUtilFilter : validateToken(jwt, userDetails)
|
||||
alt Valid token
|
||||
JwtRequestFilter -> SecurityContext : Set authentication
|
||||
else Invalid token
|
||||
JwtRequestFilter -> JwtRequestFilter : Log and skip authentication
|
||||
end
|
||||
JwtRequestFilter --> UserClient : Continue filter chain
|
||||
|
||||
== @JwtRequired Secured Method Access ==
|
||||
|
||||
box "AOP - Method Security" #E1F5FE
|
||||
participant JwtAspect
|
||||
participant JwtUtilAspect as "JwtUtil"
|
||||
end box
|
||||
|
||||
UserClient -> JwtAspect : Call method annotated with @JwtRequired
|
||||
JwtAspect -> JwtAspect : Extract JWT from header
|
||||
JwtAspect -> JwtUtilAspect : extractUsername(jwt)
|
||||
JwtAspect -> SecurityContext : Get UserDetails
|
||||
JwtAspect -> JwtUtilAspect : validateToken(jwt, userDetails)
|
||||
alt Token valid and user matches
|
||||
JwtAspect --> UserClient : Proceed to method execution
|
||||
else Invalid/mismatched token
|
||||
JwtAspect --> UserClient : Unauthorized exception
|
||||
return
|
||||
end
|
||||
|
||||
@enduml
|
||||
|
||||
** class
|
||||
@startuml
|
||||
skinparam classAttributeIconSize 0
|
||||
skinparam packageStyle rectangle
|
||||
|
||||
package "High-Level Overview" {
|
||||
|
||||
' Controller Layer
|
||||
package "Controller Layer" {
|
||||
class AiChatController
|
||||
class AiChatWebSocketController
|
||||
class AuthController
|
||||
class FileMetadataController
|
||||
class LibraryController
|
||||
class PageController
|
||||
class RegisterController
|
||||
class UserController
|
||||
class WorkspaceController
|
||||
}
|
||||
|
||||
' Service Layer
|
||||
package "Service Layer" {
|
||||
class AiChatService
|
||||
class AuthService
|
||||
class FileMetadataService
|
||||
class LibraryService
|
||||
class PageService
|
||||
class RegisterService
|
||||
class UserService
|
||||
class WorkspaceService
|
||||
}
|
||||
|
||||
' Repository Layer
|
||||
package "Repository Layer" {
|
||||
class AiChatRepository
|
||||
class FileMetadataRepository
|
||||
class LibraryRepository
|
||||
class PageRepository
|
||||
class UserRepository
|
||||
class WorkspaceRepository
|
||||
class RegisterRepository
|
||||
class AuthRepository
|
||||
}
|
||||
|
||||
' Entity Layer
|
||||
package "Entity Layer" {
|
||||
|
||||
class User {
|
||||
+id: Long
|
||||
+roles: Set<String>
|
||||
+firstName: String
|
||||
+lastName: String
|
||||
+username: String
|
||||
+password: String
|
||||
+confirmPassword: String
|
||||
+email: String
|
||||
+phoneNumber: String
|
||||
+picture: byte[]
|
||||
}
|
||||
|
||||
class Workspace {
|
||||
+id: Long
|
||||
+name: String
|
||||
+description: String
|
||||
+fileCount: int
|
||||
+noteCount: int
|
||||
+lastAccessed: LocalDateTime
|
||||
+dateCreated: LocalDateTime
|
||||
}
|
||||
|
||||
class Library {
|
||||
+id: Long
|
||||
+name: String
|
||||
+description: String
|
||||
+dateCreated: LocalDateTime
|
||||
+lastAccessed: LocalDateTime
|
||||
}
|
||||
|
||||
class Page {
|
||||
+id: Long
|
||||
+name: String
|
||||
+workspaceId: Long
|
||||
+content: Text
|
||||
+height: int
|
||||
+width: int
|
||||
}
|
||||
|
||||
class FileMetadata {
|
||||
+id: Long
|
||||
+fileUrl: String
|
||||
+userId: String
|
||||
+description: String
|
||||
+fileData: byte[]
|
||||
+fileName: String
|
||||
+fileType: String
|
||||
}
|
||||
|
||||
class AiChat {
|
||||
+id: Long
|
||||
+name: String
|
||||
+workspaceId: Long
|
||||
+createdAt: LocalDateTime
|
||||
+messages: List<AIChatMessage>
|
||||
}
|
||||
}
|
||||
|
||||
' Python NLP Microservice
|
||||
package "External NLP Module (Python FastAPI)" {
|
||||
class FastAPINLPServer {
|
||||
+/extract_text
|
||||
+/keywords
|
||||
+/tfidf_keywords
|
||||
+/summarize
|
||||
+/ner
|
||||
}
|
||||
}
|
||||
|
||||
' Entity Relationships
|
||||
User "1" --> "many" Library
|
||||
User "1" --> "many" Workspace
|
||||
Library "1" --> "many" Workspace
|
||||
Library "1" --> "many" FileMetadata
|
||||
Workspace "1" --> "many" Page
|
||||
Workspace "1" --> "many" FileMetadata
|
||||
Workspace "1" --> "many" AiChat
|
||||
|
||||
' Controller → Service
|
||||
AiChatController --> AiChatService
|
||||
AiChatWebSocketController --> AiChatService
|
||||
AuthController --> AuthService
|
||||
FileMetadataController --> FileMetadataService
|
||||
LibraryController --> LibraryService
|
||||
PageController --> PageService
|
||||
RegisterController --> RegisterService
|
||||
UserController --> UserService
|
||||
WorkspaceController --> WorkspaceService
|
||||
|
||||
' Service → Repository
|
||||
AiChatService --> AiChatRepository
|
||||
AuthService --> AuthRepository
|
||||
FileMetadataService --> FileMetadataRepository
|
||||
LibraryService --> LibraryRepository
|
||||
PageService --> PageRepository
|
||||
RegisterService --> RegisterRepository
|
||||
UserService --> UserRepository
|
||||
WorkspaceService --> WorkspaceRepository
|
||||
|
||||
' Repository → Entity
|
||||
AiChatRepository --> AiChat
|
||||
FileMetadataRepository --> FileMetadata
|
||||
LibraryRepository --> Library
|
||||
PageRepository --> Page
|
||||
UserRepository --> User
|
||||
WorkspaceRepository --> Workspace
|
||||
RegisterRepository --> User
|
||||
AuthRepository --> User
|
||||
|
||||
' Frontend → NLP Server
|
||||
|
||||
}
|
||||
|
||||
' Frontend Component
|
||||
package "Frontend" {
|
||||
class AngularClient
|
||||
}
|
||||
|
||||
' Angular communicates with both backends
|
||||
AngularClient ..> AiChatController : REST
|
||||
AngularClient ..> AiChatWebSocketController : WebSocket
|
||||
AngularClient ..> AuthController : REST
|
||||
AngularClient ..> PageController : REST
|
||||
AngularClient ..> FileMetadataController : REST
|
||||
AngularClient ..> WorkspaceController : REST
|
||||
AngularClient ..> LibraryController : REST
|
||||
AngularClient ..> RegisterController : REST
|
||||
AngularClient ..> UserController : REST
|
||||
AngularClient ..> FastAPINLPServer : REST
|
||||
|
||||
@enduml
|
||||
** client-server
|
||||
@startuml
|
||||
!pragma layout smetana
|
||||
|
||||
actor User
|
||||
|
||||
package "Client Side (Angular)" {
|
||||
[Angular App] as Frontend
|
||||
}
|
||||
|
||||
package "Spring Boot Backend" {
|
||||
[Auth Controller]
|
||||
[Note Controller]
|
||||
[NLP Proxy Service]
|
||||
[Spring Boot App] as Backend
|
||||
}
|
||||
|
||||
package "Database Layer" {
|
||||
database "PostgreSQL Database" as DB
|
||||
}
|
||||
|
||||
package "FastAPI NLP Server (Port 9010)" {
|
||||
[FastAPI NLP Service] as NLP
|
||||
[Summarisation Endpoint] as SumEndpoint
|
||||
[Keyword Extraction] as KeywordEndpoint
|
||||
[NER Endpoint] as NEREndpoint
|
||||
}
|
||||
|
||||
package "External API" {
|
||||
[OpenAI API]
|
||||
}
|
||||
|
||||
' Interactions
|
||||
User --> Frontend : Uses Web Interface
|
||||
Frontend --> Backend : REST API Calls
|
||||
Backend --> [Auth Controller] : Handles Auth
|
||||
Backend --> [Note Controller] : Handles Notes
|
||||
Backend --> [NLP Proxy Service] : Handles NLP Requests
|
||||
[Auth Controller] --> DB : Access users table
|
||||
[Note Controller] --> DB : Access notes table
|
||||
[NLP Proxy Service] --> NLP : HTTP (port 9010)
|
||||
NLP --> SumEndpoint : /summarise
|
||||
NLP --> KeywordEndpoint : /keywords
|
||||
NLP --> NEREndpoint : /ner
|
||||
SumEndpoint --> [OpenAI API] : API call for contextual help
|
||||
|
||||
note right of Frontend
|
||||
Angular app includes:
|
||||
- Note Editor
|
||||
- Summary View
|
||||
- Keyword Panel
|
||||
end note
|
||||
|
||||
note right of Backend
|
||||
Spring Boot includes:
|
||||
- Auth Controller
|
||||
- Note Controller
|
||||
- NLP Proxy Service (calls FastAPI)
|
||||
end note
|
||||
|
||||
note right of NLP
|
||||
FastAPI exposes endpoints for:
|
||||
- Summarisation (Pegasus)
|
||||
- Keyword Extraction (TF-IDF, YAKE)
|
||||
- NER (spaCy)
|
||||
end note
|
||||
@enduml
|
||||
** high level:
|
||||
@startuml
|
||||
skinparam componentStyle uml2
|
||||
skinparam packageStyle rectangle
|
||||
|
||||
package "Client (Frontend)" {
|
||||
[Angular UI]
|
||||
}
|
||||
|
||||
package "Java Backend (Spring Boot)" {
|
||||
[REST API Controller Layer]
|
||||
[WebSocket Controller]
|
||||
[Service Layer]
|
||||
[Repository Layer]
|
||||
[JPA Entities]
|
||||
}
|
||||
|
||||
package "Python NLP Service" {
|
||||
[FastAPI Server]
|
||||
[Summarizer (Pegasus)]
|
||||
[Keyword Extractor (YAKE / TF-IDF)]
|
||||
[NER (spaCy)]
|
||||
[Extract Text (Canvas Object)]
|
||||
}
|
||||
|
||||
package "External Services" {
|
||||
[OpenAI API]
|
||||
[PostgreSQL DB]
|
||||
}
|
||||
|
||||
' Communication Arrows
|
||||
[Angular UI] --> [REST API Controller Layer] : REST API (JSON)
|
||||
[Angular UI] --> [WebSocket Controller] : WebSocket
|
||||
|
||||
[REST API Controller Layer] --> [Service Layer]
|
||||
[WebSocket Controller] --> [Service Layer]
|
||||
[Service Layer] --> [Repository Layer]
|
||||
[Repository Layer] ..> [JPA Entities] : uses
|
||||
[Service Layer] --> [OpenAI API] : API calls
|
||||
|
||||
[Service Layer] --> [FastAPI Server] : REST (NLP Requests)
|
||||
[FastAPI Server] --> [Summarizer (Pegasus)]
|
||||
[FastAPI Server] --> [Keyword Extractor (YAKE / TF-IDF)]
|
||||
[FastAPI Server] --> [NER (spaCy)]
|
||||
[FastAPI Server] --> [Extract Text (Canvas Object)]
|
||||
|
||||
[Repository Layer] --> [PostgreSQL DB] : Hibernate ORM
|
||||
@enduml
|
||||
|
||||
** title AI Chat + Workspace Workflow
|
||||
@startuml
|
||||
title AI Chat + Workspace Workflow
|
||||
|
||||
skinparam shadowing true
|
||||
skinparam packageStyle rectangle
|
||||
skinparam handwritten false
|
||||
skinparam linetype polyline
|
||||
skinparam ParticipantPadding 20
|
||||
skinparam maxMessageSize 150
|
||||
|
||||
actor User
|
||||
|
||||
== AI Chat Session Creation ==
|
||||
|
||||
box "Client" #E3F2FD
|
||||
participant AngularClient as "User"
|
||||
end box
|
||||
|
||||
box "Backend - AI Chat" #E8F5E9
|
||||
participant AiChatController
|
||||
participant AiChatService
|
||||
participant AiChatRepository
|
||||
end box
|
||||
|
||||
box "External NLP Service" #FCE4EC
|
||||
participant FastAPINLPServer as "FastAPI NLP"
|
||||
end box
|
||||
|
||||
User -> AngularClient : Open AI Chat page
|
||||
AngularClient -> AiChatController : POST /ai-chat\n(chat metadata)
|
||||
AiChatController -> AiChatService : createChat(chatDto)
|
||||
AiChatService -> AiChatRepository : save(chat)
|
||||
AiChatService --> AiChatController : chatId
|
||||
AiChatController --> AngularClient : 200 OK + chatId
|
||||
|
||||
User -> AngularClient : Send message via WebSocket
|
||||
AngularClient -> AiChatWebSocketController : message
|
||||
AiChatWebSocketController -> AiChatService : handleMessage()
|
||||
|
||||
alt Message requires NLP
|
||||
AiChatService -> FastAPINLPServer : POST /summarize or /ner
|
||||
FastAPINLPServer --> AiChatService : NLP result
|
||||
end
|
||||
|
||||
AiChatService -> AiChatRepository : update(chat with message)
|
||||
AiChatWebSocketController --> AngularClient : Message response
|
||||
|
||||
== Workspace Creation ==
|
||||
|
||||
box "Backend - Workspace" #FFF3E0
|
||||
participant WorkspaceController
|
||||
participant WorkspaceService
|
||||
participant WorkspaceRepository
|
||||
end box
|
||||
|
||||
User -> AngularClient : Create Workspace
|
||||
AngularClient -> WorkspaceController : POST /workspace\n(name, description)
|
||||
WorkspaceController -> WorkspaceService : createWorkspace(dto)
|
||||
WorkspaceService -> WorkspaceRepository : save(workspace)
|
||||
WorkspaceService --> WorkspaceController : workspaceId
|
||||
WorkspaceController --> AngularClient : 200 OK + workspaceId
|
||||
|
||||
@enduml
|
||||
|
||||
* Screenshots:
|
||||
** Draft 1:
|
||||
[[file:~/master-folder/FYP/uml/draft1.png]]
|
||||
|
||||
11
Uni/20250120110833-afp.org
Normal file
11
Uni/20250120110833-afp.org
Normal file
@@ -0,0 +1,11 @@
|
||||
:PROPERTIES:
|
||||
:ID: 556d10d1-1c74-4d9f-a398-39cb3bd5d935
|
||||
:END:
|
||||
#+title: AFP (Advanced Functional Programming)
|
||||
#+filetags: :uni:index:
|
||||
|
||||
- [[id:6c272430-aff5-468b-90d3-5ff3a96152cf][AFP Week 1]]
|
||||
|
||||
- [[id:4bc71106-1d2b-4c71-836d-54b738fe5ff5][AFP Week 2]]
|
||||
|
||||
- [[id:1f395b8c-cf55-43eb-9430-dd9449f6b575][AFP Week 3]]
|
||||
10
Uni/20250120111047-afp_week1.org
Normal file
10
Uni/20250120111047-afp_week1.org
Normal file
@@ -0,0 +1,10 @@
|
||||
:PROPERTIES:
|
||||
:ID: 6c272430-aff5-468b-90d3-5ff3a96152cf
|
||||
:END:
|
||||
#+title: AFP Week 1
|
||||
#+filetags: :uni:index:
|
||||
|
||||
* <2025-01-20 Mon> : [[id:f6c9e1a7-8465-4cef-9481-2b803d0c43d4][AFP Lab 1]]
|
||||
- Installing Agda
|
||||
|
||||
* <2025-01-21 Tue> : [[id:3aef24fd-b220-4408-aa1e-c3538d661b62][AFP Lec 1]]
|
||||
115
Uni/20250120113936-afp_lab_1.org
Normal file
115
Uni/20250120113936-afp_lab_1.org
Normal file
@@ -0,0 +1,115 @@
|
||||
:PROPERTIES:
|
||||
:ID: f6c9e1a7-8465-4cef-9481-2b803d0c43d4
|
||||
:END:
|
||||
#+title: afp_lab_1
|
||||
#+filetags: :notes:uni:
|
||||
<2025-01-20 Mon>
|
||||
* Installation of Agda:
|
||||
** Step 1
|
||||
#+BEGIN_SRC eshell
|
||||
sudo apt-get install zlib1g-dev libncurses5-dev
|
||||
#+END_SRC
|
||||
|
||||
then install ghc
|
||||
- https://www.haskell.org/ghcup/
|
||||
|
||||
Now that you have cabal installed, use it to install Agda as a Haskell package:
|
||||
#+BEGIN_SRC eshell
|
||||
cabal update
|
||||
cabal install Agda
|
||||
#+END_SRC
|
||||
** Step 2
|
||||
Most users will want to install the standard library. You can install this as any other Agda library (see Library Management). See the agda-stdlib project’s installation instructions for the steps to take to install the latest version.
|
||||
https://github.com/agda/agda-stdlib
|
||||
https://agda.readthedocs.io/en/latest/tools/package-system.html#package-system
|
||||
* Exercises:
|
||||
[[file:~/master-folder/Uni/ADVFUNC/afp-learning-2024-2025/files/LectureNotes/files/exercises/my-lab1.lagda.md][my lab1]]
|
||||
|
||||
|
||||
```agda
|
||||
data Bool : Type where
|
||||
true false : Bool
|
||||
```
|
||||
|
||||
|
||||
In Agda, the snippet `data Bool : Type where true false : Bool` is a definition of a simple data type named `Bool`. Let’s break down and explain all the components and words in this snippet:
|
||||
|
||||
---
|
||||
|
||||
### 1. **`data`**
|
||||
- **What it means:** This keyword introduces a new *data type* definition in Agda.
|
||||
- **Purpose:** Defines a type that consists of one or more *constructors*, which are used to create values of the type.
|
||||
- **In this case:** The `data` keyword is used to define the `Bool` type.
|
||||
|
||||
---
|
||||
|
||||
### 2. **`Bool`**
|
||||
- **What it is:** The name of the data type being defined.
|
||||
- **Purpose:** Represents a type that has exactly two possible values: `true` and `false`.
|
||||
- **In this case:** `Bool` is a common type used to represent truth values in logic (true or false).
|
||||
|
||||
---
|
||||
|
||||
### 3. **`:` (colon)**
|
||||
- **What it means:** Indicates a type declaration in Agda.
|
||||
- **Purpose:** Separates the name of the data type (`Bool`) from its type (`Type`).
|
||||
- **In this case:** `Bool : Type` declares that `Bool` is a type itself (i.e., a type of values).
|
||||
|
||||
---
|
||||
|
||||
### 4. **`Type`**
|
||||
- **What it is:** The universe of all types in Agda.
|
||||
- **Purpose:** In Agda, types are first-class citizens, and `Type` represents the "type of types." For instance:
|
||||
- `Bool : Type` means `Bool` is a type.
|
||||
- If you define a type `ℕ : Type` for natural numbers, it also belongs to `Type`.
|
||||
- **In this case:** `Bool` is defined as a type that belongs to this universe.
|
||||
|
||||
---
|
||||
|
||||
### 5. **`where`**
|
||||
- **What it means:** Introduces the constructors (values) of the type.
|
||||
- **Purpose:** Lists the specific ways you can construct or create values of the newly defined type.
|
||||
- **In this case:** `where` introduces the two constructors, `true` and `false`, which are the only possible values of the `Bool` type.
|
||||
|
||||
---
|
||||
|
||||
### 6. **`true false`**
|
||||
- **What they are:** The *constructors* of the `Bool` type.
|
||||
- **Purpose:** Constructors are functions (or constants) that produce values of the type. Here:
|
||||
- `true : Bool` means `true` is a value of type `Bool`.
|
||||
- `false : Bool` means `false` is a value of type `Bool`.
|
||||
- **In this case:** These two constructors define all possible values of `Bool`.
|
||||
|
||||
---
|
||||
|
||||
### 7. **`Bool` after the `:`**
|
||||
- **What it means:** Specifies the type of each constructor.
|
||||
- **Purpose:** Ensures that each constructor produces a value of the correct type (`Bool` in this case).
|
||||
- **In this case:** Both `true` and `false` are of type `Bool`.
|
||||
|
||||
---
|
||||
|
||||
### Complete Meaning
|
||||
The entire definition can be read as:
|
||||
1. Define a new type called `Bool`.
|
||||
2. `Bool` belongs to the universe of types (`Type`).
|
||||
3. The `Bool` type has two constructors:
|
||||
- `true` (a value of type `Bool`).
|
||||
- `false` (a value of type `Bool`).
|
||||
|
||||
---
|
||||
|
||||
### Example Usage in Agda
|
||||
You can use `Bool` in various ways:
|
||||
```agda
|
||||
-- A function that negates a Bool
|
||||
not : Bool → Bool
|
||||
not true = false
|
||||
not false = true
|
||||
|
||||
-- A value of type Bool
|
||||
myBool : Bool
|
||||
myBool = true
|
||||
```
|
||||
|
||||
This showcases how the `Bool` type and its constructors (`true` and `false`) can be used in programs.
|
||||
154
Uni/20250121110241-afp_lec_1.org
Normal file
154
Uni/20250121110241-afp_lec_1.org
Normal file
@@ -0,0 +1,154 @@
|
||||
:PROPERTIES:
|
||||
:ID: 3aef24fd-b220-4408-aa1e-c3538d661b62
|
||||
:END:
|
||||
#+title: afp_lec_1
|
||||
#+filetags: :notes:uni:
|
||||
<2025-01-21 Tue>
|
||||
[[file:~/master-folder/Uni/ADVFUNC/afp-learning-2024-2025/files/LectureNotes/files/introduction.lagda.md][Week 1 Handout]]
|
||||
* Week 1 lec 1
|
||||
if you want to put a hole in the file (ie the { }1 ) then put a ? and load the file
|
||||
|
||||
types are default called 'sets' in agda but we will call them types.
|
||||
|
||||
data Bool : Type where
|
||||
true false : Bool
|
||||
|
||||
Here we create a type called Bool
|
||||
|
||||
--
|
||||
|
||||
data Maybe (A : Type) : Type where
|
||||
nothing : Maybe A
|
||||
just : A → Maybe A
|
||||
|
||||
Given the type A we produce another type
|
||||
Then we have two constructors, we can actually call them whatever we want, the only rule is that it shouldnt have spaces in the middle. For example, instead of nothing we can call 'nothing' as 'Nothingljfsdsdj'.
|
||||
|
||||
The below is known as coercion or the disjoint union
|
||||
Considering Blah = Either ℕ Bool
|
||||
left 0: Blah
|
||||
right false: Blah
|
||||
|
||||
--
|
||||
|
||||
data ℕ : Type where
|
||||
zero : ℕ
|
||||
suc : ℕ → ℕ
|
||||
|
||||
-- 4 = suc (suc (suc (suc zero)))
|
||||
|
||||
--
|
||||
|
||||
data List (A : Type) : Type where
|
||||
[] : List A
|
||||
_::_ : A → List A → List A
|
||||
|
||||
Below is a function that maps a type to a type:
|
||||
myList : Type -> Type
|
||||
myList = List
|
||||
|
||||
-- N -> Type ... this is called 'Dependant Type' which depends on elements of another type.
|
||||
When it comes to types, its often the case where the language we use doesnt exactly define the type we want it to. For example in haskell, the binary tree, when we want to use the binary search tree its a little difficult. In agda we can define a precise type; including the binary search tree. In summary agda can write precise types.
|
||||
|
||||
--
|
||||
|
||||
if_then_else_ : {A : Type} → Bool → A → A → A
|
||||
if true then x else y = x
|
||||
if false then x else y = y
|
||||
|
||||
this above is known as mixfix operation, we use the _ to denote where the arguments of the function are going to be places
|
||||
|
||||
--
|
||||
|
||||
_+_ : ℕ → ℕ → ℕ
|
||||
zero + y = y
|
||||
suc x + y = suc (x + y)
|
||||
|
||||
here we can pattern match on any of x or y
|
||||
in haskell, recursion can be done and there arent restrictions on it. in Agda we have structural recursion as there is a termination on recursion. Structural recursion is when you follow the structure of the definition, for example in suc x + y, we get suc (x + y), we keep removing the x from suc until we are left with 0. scrcpy
|
||||
|
||||
--
|
||||
|
||||
_*_ : ℕ → ℕ → ℕ
|
||||
zero * y = zero
|
||||
suc x * y = x * y + y
|
||||
|
||||
for suc x * y:
|
||||
(1 + x) * y
|
||||
y + x * y
|
||||
|
||||
in infixr, the r means the brackets are explicitly on the right, if we use infixl then its on the left:
|
||||
x + y + z
|
||||
x + (y + z) : r
|
||||
(x + y) + z : l
|
||||
|
||||
-- research implicit arguements.
|
||||
|
||||
|
||||
reverse : {A : Type} → List A → List A
|
||||
reverse [] = []
|
||||
reverse (x :: xs) = reverse xs ++ [ x ]
|
||||
|
||||
although this program works, its inefficient.
|
||||
|
||||
rev-append : {A : Type} → List A → List A → List A
|
||||
rev-append [] ys = ys
|
||||
rev-append (x :: xs) ys = rev-append xs (x :: ys)
|
||||
|
||||
rev : {A : Type} → List A → List A
|
||||
rev xs = rev-append xs []
|
||||
|
||||
the revappend is a helper function. you can see theres no ++ (concatenation) involved.
|
||||
* Week 1 lec 2
|
||||
|
||||
N-Induction : (P: N -> Type)
|
||||
-> P 0
|
||||
-> ((k:N) -> Pk (P (suc k))
|
||||
choose k = 0
|
||||
|
||||
P0 : P0
|
||||
f0 p1 : P1
|
||||
f1 p2 : p2
|
||||
|
||||
-> (n:N) -> Pn
|
||||
|
||||
This is proof by induction,
|
||||
|
||||
ℕ-induction P p0 f 0 = P0
|
||||
ℕ-induction P p0 f(suc n) = goal
|
||||
where
|
||||
ℍ : Pn
|
||||
ℍ = ℕ-induction P p0 f n
|
||||
|
||||
ℕ-induction P p0 f 3 =
|
||||
f3(f2(f1 p0))
|
||||
essentially this is a for loop.
|
||||
|
||||
--
|
||||
indstep : (k:N) -> k≣k -> suc k ≣ suc k
|
||||
indstep k e = e
|
||||
|
||||
ℕ-refl n = ℕ-induction (λx -> x≣x) * (λke -> e)
|
||||
|
||||
introduction and elimination rules
|
||||
|
||||
|
||||
--
|
||||
List-induction : { x : Type }
|
||||
-> (P : List X -> Type)
|
||||
-> P []
|
||||
-> ((x:X)(xs:List X) -> Pxs -> P(x::xs))
|
||||
|
||||
-> (xs : List X ) -> P xs
|
||||
|
||||
List-induction
|
||||
|
||||
|
||||
* Keybindings:
|
||||
- C-c C-l : to load the agda file
|
||||
- SPC-w for the windows
|
||||
- C-c C-c : case split (agda mode) (put x, or b or whatever the first one is, then use this - very useful).
|
||||
- C-c C-SPC: give: tries to fill the hole
|
||||
- C-shift - : undo
|
||||
- C-c C-r : refine
|
||||
- C-c C-, : show the context
|
||||
11
Uni/20250128110828-afp_week2.org
Normal file
11
Uni/20250128110828-afp_week2.org
Normal file
@@ -0,0 +1,11 @@
|
||||
:PROPERTIES:
|
||||
:ID: 4bc71106-1d2b-4c71-836d-54b738fe5ff5
|
||||
:END:
|
||||
#+title: afp_week2
|
||||
#+filetags: :uni:index:
|
||||
|
||||
<2025-01-27 Mon>
|
||||
|
||||
<2025-01-28 Tue>
|
||||
|
||||
[[id:460f4a49-8ae4-444a-bf82-4e14ca7cad3f][afp_lec_2]]
|
||||
43
Uni/20250128111008-afp_lec_2.org
Normal file
43
Uni/20250128111008-afp_lec_2.org
Normal file
@@ -0,0 +1,43 @@
|
||||
:PROPERTIES:
|
||||
:ID: 460f4a49-8ae4-444a-bf82-4e14ca7cad3f
|
||||
:END:
|
||||
#+title: afp_lec_2
|
||||
#+filetags: :uni:notes:
|
||||
|
||||
- List are dependant, thats what makes it so special.
|
||||
|
||||
|
||||
```agda
|
||||
data _≡_ {X : Type} : X → X → Type where
|
||||
refl : (x : X) → x ≡ x
|
||||
|
||||
infix 0 _≡_
|
||||
```
|
||||
Here is the identity type. For every type X, im going to define two elements of X, and the only way to define this element the constructor of which is single. The type is like a proposition and the element is like the proof. By just having this rule we can do everything we wanna do, for example, proving that if x=y, y=x.
|
||||
Definition equal : =
|
||||
Type equality : ≡
|
||||
|
||||
|
||||
|
||||
we want to write propositions as types and proofs as programs.
|
||||
|
||||
A proof is a convincing argument. How do you convince someone that something you believe is true is actually true, it ivolves reasoning.
|
||||
* AND
|
||||
To prove A and B we have to prove A, and also prove B. Proof by arguments; you have to argue/justify that A is true and that B is true.
|
||||
|
||||
a : A
|
||||
the little `a` is a justification of the the big `A`. little a is a way to prove that big A holds.
|
||||
|
||||
So for this it would be:
|
||||
a : A
|
||||
b : B
|
||||
and we need that both holds, imagine them as pairs: (a , b)
|
||||
cartesian product -> (a , b) : A x B
|
||||
the cartesian product is just a set of pairs. The below is a translation of conjunction to cartesian product in Agda.
|
||||
|
||||
```agda
|
||||
|
||||
data _ x _ (A B : Type) : Type where
|
||||
_ , _ : A -> B -> A x B
|
||||
|
||||
```
|
||||
7
Uni/20250218110239-afp_week5.org
Normal file
7
Uni/20250218110239-afp_week5.org
Normal file
@@ -0,0 +1,7 @@
|
||||
:PROPERTIES:
|
||||
:ID: 1f395b8c-cf55-43eb-9430-dd9449f6b575
|
||||
:END:
|
||||
#+title: afp_week5
|
||||
#+filetags: :uni:index:
|
||||
|
||||
[[id:ed4c372b-0314-4b6e-9119-742f69b5e434][afp_lec_5]]
|
||||
9
Uni/20250218110346-afp_lec_5.org
Normal file
9
Uni/20250218110346-afp_lec_5.org
Normal file
@@ -0,0 +1,9 @@
|
||||
:PROPERTIES:
|
||||
:ID: ed4c372b-0314-4b6e-9119-742f69b5e434
|
||||
:END:
|
||||
#+title: afp_lec_5
|
||||
#+filetags: :uni:notes:
|
||||
|
||||
*
|
||||
the first half of the lecture we went through the practice test solutions.
|
||||
for the last question we got a hint, namely to redefine the leaf and node constructors of the Rose Trees.
|
||||
43
Uni/20250329114733-ise.org
Normal file
43
Uni/20250329114733-ise.org
Normal file
@@ -0,0 +1,43 @@
|
||||
:PROPERTIES:
|
||||
:ID: c69e4c4d-2fb4-4cf1-a835-a235cf6db8e9
|
||||
:END:
|
||||
#+title: ISE
|
||||
#+filetags: :uni:index:
|
||||
|
||||
* Intelligent Software Engineering Key Revision Points (Lectures 1-7)
|
||||
** DONE Lecture 8 - 10
|
||||
Test questions will be randomly chosen from these key points. Key revision points for
|
||||
lecture 8-10 will be verbally covered during lectures
|
||||
** DONE Lecture 1
|
||||
General Bug Report Process (page 9)
|
||||
Simple Bug Report Classification Method (pages 12-20)
|
||||
** DONE Lecture 2
|
||||
DiMerent Configuration Sampling Methods (pages 8-19)
|
||||
Configuration Encodings (pages 21-28)
|
||||
Single Environment Learning (DaL) (pages 39-46)
|
||||
** DONE Lecture 3
|
||||
Classic Code Complexity Metrics (pages 8-27)
|
||||
How to Mine Bugs for Learning? (pages 37-43)
|
||||
Cross Project Prediction (HDP) (pages 50-60)
|
||||
** DONE Lecture 4
|
||||
DiMerent Coverage Metrics and Branch Concepts (pages 6-12)
|
||||
Evolutionary Algorithm (pages 24-32)
|
||||
Test Case Generation (EvoSuite) (pages 44-55)
|
||||
Multi/Many-objective Software Testing (Sapienz) (pages 80-97)
|
||||
** DONE Lecture 5
|
||||
Model free Tuning (for ORM) (pages 9-17)
|
||||
Model free Tuning (BestConfig) (pages 18-23)
|
||||
** DONE Lecture 7
|
||||
Statistical Test Selection and Use (pages 21-67)
|
||||
* Week 1
|
||||
[[id:06b2a012-4e8a-4a8a-9494-de7ed9fbe1d3][ise_week_1]]
|
||||
* Week 2
|
||||
[[id:f308642d-fcf3-410b-b154-d60582e112a2][ise_week_2]]
|
||||
* Week 3
|
||||
[[id:1d0fa257-579f-49f3-b8fd-a3b68ddccf10][ise_week_3]]
|
||||
* Week 4
|
||||
[[id:ebf874d9-0554-47f0-be8b-5c9a948738bf][ise_week_4]]
|
||||
* Week 5
|
||||
[[id:0e70c535-b145-42d9-a9ed-fe48cddbb1a5][ISE Week 5]]
|
||||
* Week 7
|
||||
[[id:9ad3f3f1-55f7-4114-bc8c-17250b6dd25d][ise_week_7]]
|
||||
56
Uni/20250329114848-ise_week_1.org
Normal file
56
Uni/20250329114848-ise_week_1.org
Normal file
@@ -0,0 +1,56 @@
|
||||
:PROPERTIES:
|
||||
:ID: 06b2a012-4e8a-4a8a-9494-de7ed9fbe1d3
|
||||
:END:
|
||||
#+title: ise_week_1
|
||||
#+filetags: :uni:notes:
|
||||
|
||||
|
||||
* Bug report classification
|
||||
|
||||
Perform bug report classification with AI following the steps
|
||||
below:
|
||||
|
||||
- Text filtering
|
||||
- Stemming
|
||||
- Indexing
|
||||
- Train machine learning models, i.e., Decision Tree, Naive Bayes, and Logistic
|
||||
Regression
|
||||
- Prediction (after data pre-processing)
|
||||
|
||||
** Text filtering:
|
||||
- Punctuation removal
|
||||
- Specific transformations occur:
|
||||
the myUser has no meaning, its just a word. You need to split it into two meaningful words.
|
||||
- Stop words should not be removed, as it changes the whole meaning of a bug report.
|
||||
- Unlike other NLP tasks, removing stop words may distort the meaning of bug reports.
|
||||
|
||||
** Stemming:
|
||||
- Stemming reduces different forms of a word to its common base by sequential application of stemming rules. For example: light caresses colours becomes Light caress colour
|
||||
|
||||
** Indexing:
|
||||
|
||||
- Indexing: we need to convert the text into numeric representation, there are two common ways of doing this: TF and IDF
|
||||
- TF: total words / specific word . for example: "bug" appears 50 times in a 150 terms.
|
||||
- IDF: We penalise certain words based on their frequency.
|
||||
- bug report classifications here often does not use IDF but TF, because some frequent terms should not be penalized
|
||||
|
||||
- For each report, we will have a vector of indexing for words in the reports according to a dictionary
|
||||
- Example: A bug report = {“this”, “bugs”, “failure”, “interesting”, …}
|
||||
Vector = <0.123, 0.34, 0.1, ….>
|
||||
|
||||
** DT
|
||||
- What is decision tree: a tree structure that split the data depending on different values.
|
||||
- Each node is a test on the attribute.
|
||||
- Each branch represents the outcome.
|
||||
|
||||
** NB
|
||||
- What is Naive Bayes? Classifier based on the conditional probability given by the Bayes theorem.
|
||||
- Calculating the probability of each class, and classify the given features into the one with higher probability.
|
||||
|
||||
** LR
|
||||
- What is logistic regression: a linear model for classification
|
||||
- Fitting the logistic growth and the sigmoid midpoint
|
||||
- Output a probability, but can use a cutoff point to decide class.
|
||||
|
||||
** Results:
|
||||
Choose the top 20/50 words when theyre used as a feature
|
||||
229
Uni/20250329121843-ise_week_2.org
Normal file
229
Uni/20250329121843-ise_week_2.org
Normal file
@@ -0,0 +1,229 @@
|
||||
:PROPERTIES:
|
||||
:ID: f308642d-fcf3-410b-b154-d60582e112a2
|
||||
:END:
|
||||
#+title: ise_week_2
|
||||
#+filetags: :uni:notes:
|
||||
|
||||
|
||||
* DONE Different Configuration Sampling Methods (pages 8-19)
|
||||
* DONE Configuration Encodings (pages 21-28)
|
||||
* DONE Single Environment Learning (DaL) (pages 39-46)
|
||||
|
||||
* 2.1
|
||||
** Configuration Sampling
|
||||
In general machine learning problem, we dont care where the data comes from , but here we do.
|
||||
|
||||
Configuration sampling is used to select representative samples for learning performance models.
|
||||
|
||||
- Types of options:
|
||||
- Binary (e.g., on/off)
|
||||
- Numeric (e.g., value ranges)
|
||||
- Goal: Balance model accuracy with sampling effort.
|
||||
|
||||
** Binary Sampling Strategies
|
||||
|
||||
*** Option-wise Strategy
|
||||
- Each binary option is selected at least once in some configuration.
|
||||
- Minimize other options to reduce unknown interaction effects.
|
||||
- Size: Linear in the number of binary options.
|
||||
|
||||
*** T-wise Strategy
|
||||
- Covers all T-wise combinations of options (T ≥ 2).
|
||||
- Example (2-wise): {001}, {010}, {100}, {111}
|
||||
- Size: Exponential in T.
|
||||
|
||||
*** Negative Option-wise Strategy
|
||||
- For each option: one configuration where it is disabled, all others enabled.
|
||||
- Adds one all-yes configuration.
|
||||
- Size: Linear.
|
||||
- Example (3 options): {110}, {101}, {011}, {111}
|
||||
- you can see the 4th one is an all-yes configuration
|
||||
|
||||
*** Random (Binary)
|
||||
- Select n configurations randomly.
|
||||
- Simple but may be less representative.
|
||||
|
||||
*** Difference Between Option-wise and Negative Option-wise Strategies
|
||||
|
||||
Both strategies are used for sampling configurations in systems with binary options, but they focus on different aspects of option selection.
|
||||
|
||||
**** Option-wise Strategy
|
||||
- **Goal:** Ensure each option is enabled (selected) at least once across configurations.
|
||||
- For every binary option, create a configuration where it is **on**.
|
||||
- Other options are minimized to avoid unknown interactions.
|
||||
- **Focus:** Testing the **presence** of each option.
|
||||
- **Example (3 options):**
|
||||
- {100} → Option 1 enabled, others off
|
||||
- {010} → Option 2 enabled, others off
|
||||
- {001} → Option 3 enabled, others off
|
||||
|
||||
**** Negative Option-wise Strategy
|
||||
- **Goal:** Ensure each option is disabled (deselected) at least once.
|
||||
- For each option, create a configuration where it is **off**, and **all others are on**.
|
||||
- Also includes a configuration where all options are **on**.
|
||||
- **Focus:** Testing the **absence** of each option.
|
||||
- **Example (3 options):**
|
||||
- {110} → Option 3 disabled
|
||||
- {101} → Option 2 disabled
|
||||
- {011} → Option 1 disabled
|
||||
- {111} → All options enabled
|
||||
|
||||
**** Comparison Summary
|
||||
|
||||
| Feature | Option-wise | Negative Option-wise |
|
||||
|-------------------------+--------------------------+-------------------------------|
|
||||
| Focus | Presence of each option | Absence of each option |
|
||||
| What is varied | Each option enabled once | Each option disabled once |
|
||||
| Other options in config | Typically disabled | Typically enabled |
|
||||
| Additional config? | Not required | Yes, includes all-on config |
|
||||
| Use case | Minimal presence testing | Influence of removing options |
|
||||
|
||||
** Non-Binary (Numeric) Sampling Strategies
|
||||
|
||||
*** One-Factor-At-A-Time (OFAT)
|
||||
- Assumes no interactions among options.
|
||||
- Varies one option at a time, others fixed at center values.
|
||||
- Size: Linear in number of options.
|
||||
- Example (values = 1,3,5): {333}, {533}, {133}, {353}, {313}, {331}, {335}
|
||||
|
||||
*** Box-Behnken Design (BBD)
|
||||
- Captures quadratic effects and 2-wise interactions.
|
||||
- Uses subset of 3^k full factorial (min, center, max).
|
||||
- Size: Exponential in number of options.
|
||||
- Example: {111}, {113}, {115}, {131}, {151}, etc.
|
||||
|
||||
*** Central Composite Design (CCD)
|
||||
- Combines:
|
||||
- 2^k factorial points
|
||||
- 2k axial points at α-distance
|
||||
- 1 center point
|
||||
- Captures curvature and interactions.
|
||||
- Example: 8 full factorial + 6 axial + {333}
|
||||
|
||||
*** Plackett-Burman Design (PBD)
|
||||
- Focus on main effects, assumes negligible interactions.
|
||||
- Uses predefined seeds, e.g., PBD(9,3)
|
||||
- First config from seed, rest by right-shifting seed.
|
||||
- Uses indices only for values.
|
||||
- Example: If O = {1,100,1000,10000,100000}, index 3 could mean 1, 1000, 100000
|
||||
|
||||
*** Random (Non-Binary)
|
||||
- Random selection of numeric configurations.
|
||||
- Risk of non-uniformity and clustering.
|
||||
- Can negatively impact learning performance.
|
||||
|
||||
** Mixed Variable Sampling
|
||||
|
||||
Some systems include both binary and non-binary (numeric) configuration options.
|
||||
These are referred to as **mixed systems**.
|
||||
|
||||
- Requires hybrid or combined strategies to ensure representative coverage.
|
||||
- One approach: **Permute over the mixed space** by combining possible binary and numeric value combinations.
|
||||
- This can grow combinatorially, so sampling techniques may be needed to reduce the total number of permutations.
|
||||
|
||||
*** Example
|
||||
- Non-binary configs: {0.1, 0.4, 5}, {0.2, 0.4, 7}, {0.2, 0.7, 5}
|
||||
- Binary configs: {1,0}, {1,1}
|
||||
- Full mixed permutations:
|
||||
- {0.1, 0.4, 5, 1, 0}
|
||||
- {0.1, 0.4, 5, 1, 1}
|
||||
- {0.2, 0.4, 7, 1, 0}
|
||||
- {0.2, 0.4, 7, 1, 1}
|
||||
- {0.2, 0.7, 5, 1, 0}
|
||||
- {0.2, 0.7, 5, 1, 1}
|
||||
|
||||
* 2.2
|
||||
** Single Environment Learning: DeepPerf
|
||||
Source: Ha & Zhang, ICSE 2019
|
||||
|
||||
DeepPerf is an early approach using deep neural networks (>3 layers) to predict software performance in configurable systems.
|
||||
|
||||
- Designed to address:
|
||||
- Small data size: Limited measurements available.
|
||||
- Feature sparsity: Only a few configuration options significantly impact performance.
|
||||
- Network instability: Tackled with tailored hyperparameter tuning.
|
||||
|
||||
** Limitation of DeepPerf
|
||||
- Does not handle sample sparsity, a major issue in configuration performance prediction.
|
||||
|
||||
** Improved Approach: Divide-and-Learn (DaL)
|
||||
Source: Gong & Chen, ESEC/FSE 2023
|
||||
|
||||
*** Key Problem: Sample Sparsity
|
||||
- Caused by:
|
||||
- Inherited feature sparsity.
|
||||
- Small configuration changes leading to drastic performance shifts.
|
||||
- Not all configurations being valid.
|
||||
- Training data is sparse due to expensive measurements.
|
||||
|
||||
*** Key Properties of Configuration Landscape
|
||||
1. Intra-division smoothness: Configurations in the same division show smooth performance variations.
|
||||
2. Inter-division sharpness: Cross-division configurations differ significantly, possibly on key options.
|
||||
|
||||
Risk: Limited data might lead to overfitting within divisions.
|
||||
|
||||
*** Architecture of DaL
|
||||
|
||||
Three Goals:
|
||||
1. Divide the configuration data into meaningful divisions → function ϕ
|
||||
2. Learn a local model for each division → function μ
|
||||
3. Assign new configurations to the correct local model → using ϕ and μ
|
||||
|
||||
- Implementation:
|
||||
- CART (Decision Tree) is used for dividing.
|
||||
- DeepPerf models are trained within each division.
|
||||
- Random Forest is used for classifying unseen configurations into divisions.
|
||||
|
||||
*** Trade-off: Number of Divisions
|
||||
- More divisions → better at tackling sparsity, but less data per model → risks underfitting.
|
||||
- Need to balance:
|
||||
- Generalizability vs.
|
||||
- Specialization
|
||||
|
||||
*** Results
|
||||
- DaL outperforms or matches state-of-the-art in 33 out of 40 cases.
|
||||
- Achieves up to 1.94× improvement.
|
||||
- Needs fewer training samples for same accuracy.
|
||||
- Especially beneficial in complex systems or with more training data.
|
||||
* 2.3
|
||||
** Single Environment Learning: Encoding
|
||||
Source: Gong & Chen, MSR 2022
|
||||
|
||||
A study conducted by the lab investigates how different encoding schemes impact the software performance learning pipeline.
|
||||
|
||||
*** Three Common Encoding Schemes
|
||||
- Label encoding
|
||||
- Scaled label encoding (e.g., max-min normalization)
|
||||
- One-hot encoding
|
||||
|
||||
** Encoding Schemes Explained
|
||||
|
||||
*** Label Encoding
|
||||
- Converts configuration options into numeric values.
|
||||
- Example:
|
||||
- Configuration: (cache_size, interval, ssl, data_strategy)
|
||||
- Values: cache_size = (1, 10, 10000), interval = (1–4), ssl = (0, 1), data_strategy = (strategy_1, strategy_2, strategy_3)
|
||||
- Encoded: (10000, 2, 1, 1) → (2, 1, 1, 1) → data_strategy: (0, 1, 2)
|
||||
|
||||
*** Scaled Label Encoding
|
||||
- Similar to label encoding but normalizes all values to the range [0, 1].
|
||||
- Example (10000, 2, 1, 1) becomes (1, 1/3, 1, 0.5)
|
||||
|
||||
*** One-Hot Encoding
|
||||
- Transforms each categorical value into a binary vector.
|
||||
- Example: (10000, 2, 1, 1) becomes (0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 0)
|
||||
|
||||
** Community Debate and Justifications
|
||||
|
||||
- Categorical features (e.g., cache_mode = memory, disk, mixed):
|
||||
- Label encoding implies false ordering (1, 2, 3)
|
||||
- One-hot encoding avoids this but may introduce multicollinearity.
|
||||
|
||||
- Numeric options (e.g., cache_size = 1, 10, 10000):
|
||||
- Label encoding maintains order but struggles with large scale differences.
|
||||
- Scaled label encoding improves numeric stability but weakens interaction with binary features.
|
||||
|
||||
** Study Protocol
|
||||
- Evaluated using 7 learning algorithms across 5 software systems.
|
||||
|
||||
|
||||
182
Uni/20250329142725-ise_week_3.org
Normal file
182
Uni/20250329142725-ise_week_3.org
Normal file
@@ -0,0 +1,182 @@
|
||||
:PROPERTIES:
|
||||
:ID: 1d0fa257-579f-49f3-b8fd-a3b68ddccf10
|
||||
:END:
|
||||
#+title: ise_week_3
|
||||
#+filetags: :uni:notes:
|
||||
|
||||
|
||||
* DONE Classic Code Complexity Metrics (pages 8-27)
|
||||
* DONE How to Mine Bugs for Learning? (pages 37-43)
|
||||
* DONE Cross Project Prediction (HDP) (pages 50-60)
|
||||
|
||||
* 3.1 Intelligent Software Engineering: Classic Metrics
|
||||
** Software Defect Prediction
|
||||
- The foundation of software defect prediction lies in metric identification.
|
||||
- This was a key research direction in the 1980s.
|
||||
- Metrics aim to quantify properties of code to detect potential defects and improve quality.
|
||||
** Classic Code Metrics
|
||||
|
||||
*** 1. McCabe Cyclomatic Complexity
|
||||
- Purpose: Measures the complexity of code based on the number of linearly independent paths in the code’s flow graph.
|
||||
|
||||
**** Why it Matters
|
||||
- More conditional statements = More possible execution paths = Higher complexity.
|
||||
- Useful for identifying complex, hard-to-test, and error-prone code.
|
||||
|
||||
**** Simple Definition
|
||||
- McCabe Complexity = Number of simple conditions + 1
|
||||
|
||||
**** What is a “Simple Condition”?
|
||||
- A conditional without logical connectors (AND, OR).
|
||||
- Examples:
|
||||
- if (a > b)
|
||||
- while (a > b)
|
||||
- for (a=b; a > b; b++)
|
||||
- do {…} while (a > b)
|
||||
|
||||
**** Compound Conditions
|
||||
- Count each simple condition inside:
|
||||
- if (a > b || a > 2) → 2 simple conditions
|
||||
- if (a > b && a > 2) → 2 simple conditions
|
||||
|
||||
**** Use Case
|
||||
- Helps determine test case count needed for complete branch coverage.
|
||||
|
||||
*** 2. Halstead Complexity Measures
|
||||
- Purpose: Measures complexity based on the operators and operands used in code.
|
||||
|
||||
**** Definitions:
|
||||
- n1: Number of distinct operators (e.g., !=, !, %, /, *, +, &&, ||)
|
||||
- n2: Number of distinct operands (e.g., variable names, constants, types like bool, char)
|
||||
- N1: Total occurrences of operators
|
||||
- N2: Total occurrences of operands
|
||||
|
||||
**** Why Use Halstead?
|
||||
- Evaluates:
|
||||
- Code length
|
||||
- Code vocabulary
|
||||
- Effort required to implement or understand the code
|
||||
- Potential bugs
|
||||
|
||||
*** 3. Lines of Code (LOC)
|
||||
- LOC: Total number of lines in a program.
|
||||
- Comment Lines: Lines containing only comments.
|
||||
|
||||
**** Usefulness:
|
||||
- Simple indicator of:
|
||||
- Code size
|
||||
- Code density
|
||||
- Maintainability and readability
|
||||
|
||||
* 3.2 Intelligent Software Engineering: Within-Project Prediction
|
||||
** Just-in-Time (JIT) Defect Prediction
|
||||
- Based on the classic work by Kim et al. (2008).
|
||||
- Focuses on predicting defects at the commit/change level rather than file or module level.
|
||||
|
||||
** Steps in the JIT Defect Prediction Pipeline:
|
||||
1. File-level changes are extracted from a project's revision history.
|
||||
2. Bug fix changes are identified using keywords in SCM (Source Code Management) change log messages.
|
||||
3. Bug-introducing and clean changes are identified by tracing backwards from the bug fix commits.
|
||||
4. A classification model (e.g., SVM) is trained on these labeled examples.
|
||||
5. Once trained, the classifier can predict if new code changes are likely to be buggy or clean.
|
||||
|
||||
** Change-wise Prediction Details
|
||||
|
||||
*** Change History Extraction
|
||||
- Collected information includes:
|
||||
- Change log
|
||||
- Author
|
||||
- Change date
|
||||
- Source code
|
||||
- Change delta
|
||||
- Change metadata
|
||||
|
||||
*** Identifying Bug-Introducing Changes
|
||||
|
||||
**** Step 1: Search for Bug Fixes
|
||||
- Use keywords (e.g., “fix”, “bug”, “patch”) to find bug-fixing commits.
|
||||
|
||||
**** Step 2: Use the SZZ Algorithm
|
||||
- Determine what was changed in bug fixes.
|
||||
- Produces a list of regions ("hunks") showing differences between two revisions.
|
||||
- Deleted or modified code in each hunk is treated as the location of a bug.
|
||||
- Traces origin of this code to find the earlier bug-introducing changes.
|
||||
|
||||
** Example Walkthrough
|
||||
|
||||
*** Revision 1:
|
||||
- Initial creation of a function `bar`.
|
||||
- Introduces a bug: `if (report == null)` (should be `!=`).
|
||||
- SCM annotate shows all lines as modified in revision 1 by "kim".
|
||||
|
||||
*** Revision 2:
|
||||
- Two changes:
|
||||
- Function `bar` renamed to `foo`.
|
||||
- Argument changed from `report` to `report.str` in `println`.
|
||||
- Annotate output shows lines 1 and 4 were last modified by "ejw" in revision 2.
|
||||
|
||||
*** Revision 3:
|
||||
- Bug fix applied: changes `==` to `!=` on line 3.
|
||||
- SZZ algorithm compares revisions 3 and 2, identifying line 3 as modified.
|
||||
- Traces line 3’s origin back to revision 1 — identifying the bug-introducing change.
|
||||
|
||||
* 3.3 Intelligent Software Engineering: Cross-Project Prediction (HDP)
|
||||
** Heterogeneous Defect Prediction (HDP)
|
||||
- Based on the work by Nam and Kim (2015).
|
||||
- Motivation: Metrics used for defect prediction often differ across projects.
|
||||
- Goal: Address the metric mismatching problem across projects (heterogeneous settings).
|
||||
- Classifier agnostic — can be used with any machine learning model.
|
||||
|
||||
** HDP Architecture
|
||||
|
||||
*** Metric Selection in Source Datasets
|
||||
- Uses well-known feature selection methods:
|
||||
- Gain ratio
|
||||
- Chi-square
|
||||
- Relief-F
|
||||
- Significance attribute evaluation
|
||||
- Empirical testing used to choose the best approach.
|
||||
- Top 15% metrics per source project are selected.
|
||||
- Metric mismatching arises because each project may prioritize different metrics.
|
||||
|
||||
*** Matching Source and Target Metrics
|
||||
|
||||
**** Key Steps:
|
||||
1. Pair all metrics from source and target projects.
|
||||
2. Remove poorly matched metrics based on a cutoff threshold for matching scores.
|
||||
3. Apply maximum weighted bipartite matching to select the best group of matched metric pairs:
|
||||
- Goal: Maximize sum of matching scores.
|
||||
- Ensure no duplicated metrics are selected.
|
||||
|
||||
**** Example:
|
||||
- 2 source metrics: X1, X2
|
||||
- 2 target metrics: Y1, Y2
|
||||
- Matching pairs: (X1,Y1), (X1,Y2), (X2,Y1), (X2,Y2)
|
||||
|
||||
After applying a cutoff threshold of 0.30:
|
||||
- Group 1: (X1,Y1) and (X2,Y2) with total score 1.3 (=0.8+0.5)
|
||||
- Group 2: (X2,Y1) with score 0.4 (Stands alone (can't be paired with any other remaining pair without duplication)).
|
||||
- Group 1 is chosen as the matched metric set.
|
||||
|
||||
** Methods for Calculating Matching Scores
|
||||
|
||||
*** Percentile-Based Method
|
||||
- Compares 9 percentiles (10th, 20th, ..., 90th) between source and target metric values.
|
||||
- Uses the formula:
|
||||
Pij(n) = 1 - |spij(n) - bpij(n)| / bpij(n)
|
||||
- spij(n): smaller percentile value
|
||||
- bpij(n): bigger percentile value
|
||||
- Matching score is 1 when all percentiles are identical.
|
||||
|
||||
*** Kolmogorov-Smirnov (KS) Test Method
|
||||
- Non-parametric two-sample test.
|
||||
- Useful when distributions are unknown or have unequal variances.
|
||||
- Computes a p-value to indicate the similarity.
|
||||
- Matching score derived from the p-value.
|
||||
|
||||
*** Spearman’s Rank Correlation Coefficient Method
|
||||
- Measures correlation between two sets of values.
|
||||
- If dataset sizes differ, randomly sample the larger set to match sizes.
|
||||
|
||||
** Classifier Independence
|
||||
- HDP approach can be paired with any machine learning algorithm (e.g., SVM, RF, etc.)
|
||||
268
Uni/20250329200158-ise_week_4.org
Normal file
268
Uni/20250329200158-ise_week_4.org
Normal file
@@ -0,0 +1,268 @@
|
||||
:PROPERTIES:
|
||||
:ID: ebf874d9-0554-47f0-be8b-5c9a948738bf
|
||||
:END:
|
||||
#+title: ise_week_4
|
||||
#+filetags: :uni:notes:
|
||||
|
||||
|
||||
|
||||
* DONE Different Coverage Metrics and Branch Concepts (pages 6-12)
|
||||
* DONE Evolutionary Algorithm (pages 24-32)
|
||||
* DONE Test Case Generation (EvoSuite) (pages 44-55)
|
||||
* DONE Multi/Many-objective Software Testing (Sapienz) (pages 80-97)
|
||||
|
||||
want to figure out how to translate the real world phenotype to a genotype
|
||||
* 4.2 Evolutionary Algorithms - Intelligent Software Engineering
|
||||
|
||||
** 1. Illustrative Optimization Problem
|
||||
|
||||
- Problem: Maximize the objective function \( f(x) = x^2 \)
|
||||
- Design variable: \( x \in \{-15, -14, ..., 0, 1, ..., 15\} \)
|
||||
- Search space: All integers between -15 and 15 inclusive
|
||||
- Objective function: \( f(x) = x^2 \), to be maximized
|
||||
- Constraints: None
|
||||
- This problem is simple and allows us to demonstrate the application of evolutionary algorithms without involving additional complexity from constraints.
|
||||
|
||||
** 2. Representation
|
||||
|
||||
Evolutionary algorithms operate on representations of solutions called genotypes, which map to actual solutions (phenotypes). The choice of representation is crucial and problem-dependent.
|
||||
|
||||
*** Binary Representation
|
||||
|
||||
- The solution is represented as a fixed-length binary string.
|
||||
- For the example problem (maximizing \( f(x) = x^2 \)), we use a 5-bit binary representation:
|
||||
- The first bit indicates the sign of \( x \): 0 for positive, 1 for negative.
|
||||
- The remaining bits represent the magnitude in binary.
|
||||
- The genotype space is \( \{0,1\}^L \), where L is the length of the binary string.
|
||||
|
||||
*** Other Common Representations
|
||||
|
||||
- **Binary**: Suitable for many simple problems.
|
||||
- **Integer**: Useful for categorical or discrete variables (e.g., car brands such as Toyota, Volkswagen, etc.).
|
||||
- **Floating Point**: Used for problems with continuous variables. For example, optimizing \( f(x_1, x_2) = x_1 + x_2 \), where \( x_1, x_2 \in [0,1] \).
|
||||
- **Permutations**: Suitable for ordering problems like the Traveling Salesman Problem.
|
||||
- **Matrices**: Employed in more complex problems such as staff allocation or scheduling.
|
||||
|
||||
** 3. Evolutionary Algorithm Steps
|
||||
|
||||
The typical steps in an evolutionary algorithm include:
|
||||
|
||||
1. **Initialization**:
|
||||
- Start with a randomly generated population of candidate solutions.
|
||||
- Ensure a diverse set of individuals to explore the search space effectively.
|
||||
- Optionally include known solutions or use heuristics to seed the initial population.
|
||||
|
||||
2. **Evaluation**:
|
||||
- Each individual is evaluated using a fitness function.
|
||||
- The fitness function quantifies how well an individual performs with respect to the problem objective.
|
||||
|
||||
3. **Main Loop** (repeats until termination condition is met):
|
||||
a. **Selection**:
|
||||
- Select parent individuals based on their fitness.
|
||||
- Higher fitness individuals have a higher chance of being selected.
|
||||
|
||||
b. **Recombination (Crossover)**:
|
||||
- Combine selected parents to produce new offspring.
|
||||
- Occurs with probability \( P_c \) (crossover probability).
|
||||
|
||||
c. **Mutation**:
|
||||
- Randomly alter offspring genes to maintain diversity.
|
||||
- Occurs with probability \( P_m \) (mutation probability).
|
||||
|
||||
d. **Evaluation of Offspring**:
|
||||
- Assess the fitness of each newly created individual.
|
||||
|
||||
e. **Survivor Selection**:
|
||||
- Decide which individuals (from parents and offspring) will make up the next generation.
|
||||
- Can use various strategies like elitism or generational replacement.
|
||||
|
||||
** 4. Fitness Function
|
||||
|
||||
- The fitness function is derived from the problem’s objective or quality function.
|
||||
- It assigns a single real-valued score to each individual (phenotype).
|
||||
- The function reflects the degree to which a solution meets the desired criteria.
|
||||
- Typically, the aim is to **maximize** fitness.
|
||||
- If the problem is better posed as a minimization task, it can be transformed accordingly (e.g., minimizing \( f(x) \) is equivalent to maximizing \( -f(x) \)).
|
||||
|
||||
* 4.3 Test Case Generation using EvoSuite - Intelligent Software Engineering
|
||||
|
||||
** 1. Introduction to EvoSuite
|
||||
|
||||
- EvoSuite is a tool developed by Fraser and Arcuri (2011) for automated test case generation.
|
||||
- It generates whole test suites (not just individual test cases) for a given software system.
|
||||
- The tool accepts a list of input classes to be tested and produces corresponding JUnit test case code.
|
||||
- It leverages **genetic algorithms**, a form of evolutionary computation, to evolve effective test suites.
|
||||
|
||||
** 2. Motivation and Limitations of Traditional Methods
|
||||
|
||||
- Conventional test generation tools typically focus on **single coverage goals** (e.g., a single program branch).
|
||||
- Assumes:
|
||||
- All coverage goals are equally important.
|
||||
- All goals are equally difficult to reach.
|
||||
- Goals are independent of each other.
|
||||
- These assumptions are problematic:
|
||||
- The sequence in which goals are selected can significantly affect the quality of the resulting test suite.
|
||||
- Interdependencies among goals are often ignored.
|
||||
|
||||
*** Solution:
|
||||
- Generate **whole test suites** rather than isolated test cases.
|
||||
- Takes into account relationships between methods/classes.
|
||||
|
||||
** 3. Architecture and Representation
|
||||
|
||||
*** Test Suite Representation
|
||||
|
||||
- A test suite \( T \) consists of multiple test cases.
|
||||
- Each **test case** is a sequence of statements of varying types and lengths.
|
||||
- The total length of a test suite is the sum of the lengths of its individual test cases.
|
||||
|
||||
*** Statement Types in Test Cases
|
||||
|
||||
1. **Primitive statements**: Initialize basic types (e.g., `int var0 = 54`)
|
||||
2. **Constructor statements**: Create new instances (e.g., `Stack var1 = new Stack()`)
|
||||
3. **Field statements**: Access object members (e.g., `int var2 = var1.size`)
|
||||
4. **Method statements**: Call methods (e.g., `int var3 = var1.pop()`)
|
||||
|
||||
** 4. Fitness Function
|
||||
|
||||
- Guides the **selection of parents** in the genetic algorithm.
|
||||
- Aims to **maximize code coverage**.
|
||||
- If two test suites achieve the same coverage, the one with fewer statements is preferred (parsimony).
|
||||
- Uses **branch coverage** as the primary metric.
|
||||
- Employs the **branch distance heuristic**:
|
||||
- Measures how close an input is to flipping a predicate’s boolean outcome.
|
||||
|
||||
** 5. Bloat Control
|
||||
|
||||
- A known issue in Genetic Algorithms is **bloat**, where test cases grow unnecessarily large.
|
||||
- Can lead to memory exhaustion and inefficiency.
|
||||
|
||||
*** Techniques Used:
|
||||
- Set limits:
|
||||
- Maximum number of test cases \( N \)
|
||||
- Maximum length per test case \( L \)
|
||||
- Discard offspring that do not provide improved coverage.
|
||||
|
||||
** 6. Search Operators
|
||||
|
||||
*** Crossover Operator
|
||||
|
||||
- Combines two parent test suites (P1 and P2) to generate two offspring (O1 and O2).
|
||||
- O1 = first \( a \cdot |P1| \) test cases from P1 + remaining from P2.
|
||||
- O2 = similar combination from P2 and P1.
|
||||
- Valid since test cases are independent.
|
||||
- Helps reduce difference in length between resulting test suites.
|
||||
|
||||
*** Mutation Operator
|
||||
|
||||
- Mutation is applied with a probability of \( 1/T \), where \( T \) is the number of test cases.
|
||||
- New test cases may be added with a probability \( p \), up to a maximum count \( N \).
|
||||
|
||||
*** Mutation Operations (applied with equal probability 1/3):
|
||||
|
||||
1. **Remove**:
|
||||
- Each statement \( s_i \) is deleted with probability \( 1/n \), where \( n \) is the number of statements.
|
||||
- If needed, replace deleted statements to keep test case valid.
|
||||
|
||||
2. **Change**:
|
||||
- Each statement \( s_i \) may be altered.
|
||||
- For primitives: change numeric value randomly within ±Δ.
|
||||
- For others: change to a method/field/constructor of the same type.
|
||||
|
||||
3. **Insert**:
|
||||
- A new statement is inserted at a random position in the test case.
|
||||
|
||||
** 7. Results and Evaluation
|
||||
|
||||
- Key takeaway: **EvoSuite outperforms traditional single-goal test generation tools**.
|
||||
- Reported improvement: Up to **18x better branch coverage** than single-branch strategies.
|
||||
|
||||
|
||||
* 4.4 Multi/Many-objective Software Testing with Sapienz - Intelligent Software Engineering
|
||||
|
||||
** 1. Introduction to Sapienz
|
||||
|
||||
- Sapienz is an automated software testing tool developed by Mao et al. (2016).
|
||||
- It uses evolutionary algorithms to generate test cases for Android apps.
|
||||
- Notable Achievements:
|
||||
- Tested the top 1000 most popular Google Play apps.
|
||||
- Discovered 558 unique and previously unknown app crashes.
|
||||
- Led to a commercial spinout company named **MaJiCkE**.
|
||||
- Acquired by **Facebook/Meta**.
|
||||
|
||||
- Sapienz customizes the **NSGA-II** algorithm (a multi-objective genetic algorithm) for test case generation.
|
||||
|
||||
Reference: Mao, Ke, Mark Harman, and Yue Jia. *"Sapienz: Multi-objective automated testing for android applications."* ISSTA 2016.
|
||||
|
||||
** 2. NSGA-II: An Overview
|
||||
|
||||
- NSGA-II is a Genetic Algorithm (GA) adapted for **multi-objective optimization**.
|
||||
- Key Differences from standard GA:
|
||||
- Uses **Pareto dominance** for survival selection.
|
||||
- A solution **a dominates** solution **b** if:
|
||||
- \( a_i \leq b_i \) for all objectives, and
|
||||
- \( \exists j \) such that \( a_j < b_j \)
|
||||
- A **Pareto optimal** solution is one that is not dominated by any other in the population.
|
||||
|
||||
- NSGA-II also uses:
|
||||
- **Non-dominated sorting**: Separates population into Pareto fronts.
|
||||
- **Crowding distance**: Prefers diverse solutions within the same front.
|
||||
|
||||
** 3. Representation
|
||||
|
||||
- Specific representation details were not included in the slides but are tailored to represent Android GUI interaction sequences.
|
||||
|
||||
** 4. Objective Functions in Sapienz
|
||||
|
||||
Sapienz optimizes multiple objectives simultaneously:
|
||||
|
||||
*** a. Code Coverage
|
||||
|
||||
- Types of coverage used:
|
||||
- **Statement Coverage**: Measures how many individual code statements are executed.
|
||||
- **Method Coverage**: Measures the number of methods invoked.
|
||||
- **Android Activity Coverage**: Tracks which screens (activities) of the app are accessed.
|
||||
- Example: A dialer app may include separate activities for contacts, keypad, call history, etc.
|
||||
|
||||
*** b. Test Case Length
|
||||
|
||||
- Shorter test cases are generally preferred to improve efficiency and reduce overhead.
|
||||
- Multiple slides (86–89) emphasize the importance of minimizing test length.
|
||||
|
||||
*** c. Crash Discovery
|
||||
|
||||
- The number of test cases that lead to app crashes is also a key metric.
|
||||
- Objective: Maximize the number of crash-inducing test cases.
|
||||
|
||||
** 5. Search Operators
|
||||
|
||||
*** a. Crossover
|
||||
|
||||
- Combines parts of two parent test sequences to form new offspring.
|
||||
- Details of the crossover structure are tool-specific but follow the typical GA-style recombination.
|
||||
|
||||
*** b. Mutation
|
||||
|
||||
- Mutations are applied to test cases to explore new behaviors.
|
||||
- Types of Mutation:
|
||||
|
||||
1. **High-Level Mutation**:
|
||||
- Alters the structure or intent of test sequences.
|
||||
|
||||
2. **Low-Level Mutation (Same Size)**:
|
||||
- Changes test actions without altering the sequence length.
|
||||
|
||||
3. **Low-Level Mutation (Different Size)**:
|
||||
- Adds or removes actions to vary the length of test cases.
|
||||
|
||||
4. **Low-Level Mutation (Shuffling)**:
|
||||
- Reorders existing actions in the test case.
|
||||
|
||||
** 6. Results and Observations
|
||||
|
||||
- Sapienz significantly **outperforms other automated testing tools** in terms of:
|
||||
- Number of crashes detected.
|
||||
- Coverage achieved.
|
||||
- Efficiency in test generation.
|
||||
|
||||
|
||||
117
Uni/20250331201944-ise_week_5.org
Normal file
117
Uni/20250331201944-ise_week_5.org
Normal file
@@ -0,0 +1,117 @@
|
||||
:PROPERTIES:
|
||||
:ID: 0e70c535-b145-42d9-a9ed-fe48cddbb1a5
|
||||
:END:
|
||||
#+title: ISE Week 5
|
||||
#+filetags: :uni:notes:
|
||||
|
||||
* TODO Model free Tuning (for ORM) (pages 9-17)
|
||||
* TODO Model free Tuning (BestConfig) (pages 18-23)
|
||||
|
||||
|
||||
* 5.1 Model-Free Tuning for ORM Systems - Intelligent Software Engineering
|
||||
|
||||
** 1. Introduction and Background
|
||||
|
||||
- This approach was proposed by Singh et al. (2016).
|
||||
- The goal is to optimize Object-Relational Mapping (ORM) systems without relying on models.
|
||||
- Uses **NSGA-II**, a multi-objective evolutionary algorithm, to handle **multiple performance concerns**.
|
||||
|
||||
Reference: Singh, Ravjot et al. *"Optimizing the performance-related configurations of object-relational mapping frameworks using a multi-objective genetic algorithm."* ACM/SPEC ICPE 2016.
|
||||
|
||||
** 2. Architecture and Setup
|
||||
|
||||
- Focuses exclusively on **binary configuration options**.
|
||||
- Example of a configuration: `{0011}` – a binary vector where each bit represents a configuration toggle (on/off).
|
||||
- Evaluates performance using three objective metrics:
|
||||
- **Execution time**
|
||||
- **CPU load**
|
||||
- **Memory consumption**
|
||||
- Other components follow the standard NSGA-II flow: selection, crossover, mutation, and fitness evaluation.
|
||||
|
||||
** 3. Stopping Criteria
|
||||
|
||||
Two specific stopping rules are proposed for determining when to terminate the evolutionary process:
|
||||
|
||||
*** a. Setting 1: t-test-based Stopping
|
||||
|
||||
- Conducts statistical **t-tests** to compare changes in objective values between generations.
|
||||
- For each pair of consecutive generations \( g_i \) and \( g_j \):
|
||||
- Run a t-test on ∆CPU and ∆MEM between all configurations in both generations.
|
||||
- If **all p-values > 0.05** for **two consecutive generations**, it indicates **no statistically significant improvement**, and the algorithm is stopped.
|
||||
|
||||
*** b. Setting 2: Mutual Dominance Rate (MDR)
|
||||
|
||||
- Measures how much progress is made by comparing the current and previous generation.
|
||||
- Let:
|
||||
- Set A = configurations from the previous generation
|
||||
- Set B = configurations from the current generation
|
||||
- Define \( dom(A,B) \) as the number of configurations in A that are **dominated** by any configuration in B.
|
||||
|
||||
Interpretation:
|
||||
- **MDR = 0**: No progress — performance plateau
|
||||
- **MDR < 0**: Regression — performance is deteriorating
|
||||
- **MDR > 0**: Improvement — current generation is better than the last
|
||||
|
||||
** 4. Termination Condition
|
||||
|
||||
- The tuning process should stop if **any** of the defined stopping conditions (t-test or MDR) are met.
|
||||
|
||||
** 5. Experimental Results
|
||||
|
||||
- NSGA-II consistently found configurations that ranked within the **top 25%** of all possible configurations across tested applications.
|
||||
- Results were obtained by combining different **aggregation functions** and **stopping rules**, demonstrating strong generalization and effectiveness.
|
||||
|
||||
|
||||
* 5.2 Model-Free Tuning with BestConfig - Intelligent Software Engineering
|
||||
|
||||
** 1. Introduction and Background
|
||||
|
||||
- BestConfig is a model-free configuration tuning system proposed by **Zhu et al. (2017)**.
|
||||
- It focuses on tuning for **a single performance objective** (e.g., throughput, latency).
|
||||
- Utilizes **local search techniques** rather than global evolutionary approaches.
|
||||
- Employs **label encoding** for parameters (e.g., {0, 23, 100}).
|
||||
- Key strategy: aggressively explore **promising regions** of the configuration space.
|
||||
|
||||
Reference: Zhu, Yuqing et al. *"BestConfig: tapping the performance potential of systems via automatic configuration tuning."* SoCC 2017.
|
||||
|
||||
** 2. Architecture Overview
|
||||
|
||||
- BestConfig is designed to intelligently search a high-dimensional configuration space.
|
||||
- Architecture relies on two core components:
|
||||
- DDS (Divide & Diverge Sampling)
|
||||
- RBS (Recursive Bound & Search)
|
||||
|
||||
** 3. DDS: Divide & Diverge Sampling
|
||||
|
||||
- Purpose: Ensures **coverage** of the entire configuration space by dividing it into **subspaces**.
|
||||
- Process:
|
||||
1. Each configuration parameter's range is divided into **k intervals**.
|
||||
2. These intervals are combined across all parameters, forming \( k^n \) subspaces.
|
||||
3. **One random sample** is taken from each subspace.
|
||||
|
||||
- Advantages:
|
||||
- Avoids bias in sampling (common in uniform random search).
|
||||
- More likely to sample from all areas of the space.
|
||||
- Especially useful in **high-dimensional spaces**.
|
||||
|
||||
** 4. RBS: Recursive Bound & Search
|
||||
|
||||
- Purpose: Locally refines and improves the best-known configuration.
|
||||
- Steps:
|
||||
1. Identify the best-performing configuration \( C_0 \) from the initial samples.
|
||||
2. Define **bounds** for each parameter based on neighboring values around \( C_0 \).
|
||||
3. Sample new points within this bounded space to find a better configuration \( C_1 \).
|
||||
4. Repeat the bounding and sampling process **recursively** until no improvement is found.
|
||||
|
||||
- Bound Definition:
|
||||
- For each parameter value in \( C_0 \), the closest lower and higher values in the dataset are chosen as bounds.
|
||||
|
||||
- Termination Conditions:
|
||||
- If no better configuration is found in a recursive round, the search **restarts from a broader space**.
|
||||
- The entire tuning process **stops** only when a **predefined resource budget** (e.g., time, evaluations) is exhausted.
|
||||
|
||||
** 5. Results and Observations
|
||||
|
||||
- BestConfig consistently finds configurations **significantly better than the system’s default settings**.
|
||||
- Achieves these improvements within a **reasonable time frame**, making it practical for real-world use.
|
||||
|
||||
134
Uni/20250331202447-ise_week_7.org
Normal file
134
Uni/20250331202447-ise_week_7.org
Normal file
@@ -0,0 +1,134 @@
|
||||
:PROPERTIES:
|
||||
:ID: 9ad3f3f1-55f7-4114-bc8c-17250b6dd25d
|
||||
:END:
|
||||
#+title: ise_week_7
|
||||
#+filetags: :uni:notes:
|
||||
|
||||
* DONE Statistical Test Selection and Use (pages 21-67)
|
||||
|
||||
* 7.1 Comparing Algorithms in Intelligent Software Engineering
|
||||
|
||||
** 1. Motivation
|
||||
|
||||
- Algorithms and configurations vary in performance.
|
||||
- No universally best algorithm: performance depends on the specific problem ("No Free Lunch" theorem).
|
||||
- To determine which algorithm/configuration is suitable, **comparison is essential**.
|
||||
- However, comparison is challenging due to the **stochastic nature** of computational intelligence algorithms.
|
||||
|
||||
** 2. Stochastic Behaviour in Algorithms
|
||||
|
||||
- Sources of randomness:
|
||||
- In the algorithm (e.g., random initial population, stochastic gradient descent, mutation/crossover probabilities).
|
||||
- In data sampling.
|
||||
- Result: Running the same algorithm multiple times on the same problem yields **different results**.
|
||||
- Therefore, comparisons must account for this randomness.
|
||||
|
||||
** 3. Handling Stochastic Behaviour
|
||||
|
||||
- To compare algorithms meaningfully:
|
||||
- Run each algorithm **multiple times** (e.g., 30+ runs) using **different random seeds**.
|
||||
- This helps capture typical performance and reduce reliance on single-run outliers.
|
||||
|
||||
** 4. Methods for Comparison
|
||||
|
||||
*** a. Mean (Average)
|
||||
|
||||
- Simple and common.
|
||||
- Problems:
|
||||
- Sensitive to **outliers**.
|
||||
- Does not represent **variability** in results.
|
||||
|
||||
*** b. Mean + Standard Deviation
|
||||
|
||||
- Adds information about variation.
|
||||
- Still affected by outliers.
|
||||
- Hard to tell whether differences are statistically significant.
|
||||
|
||||
*** c. Median
|
||||
|
||||
- More robust to outliers.
|
||||
- Example:
|
||||
- Sorted list: 0.000001, 0.6, 0.62, 0.65, 0.7, 0.75, 0.8, 0.8, 0.81
|
||||
- Median = 0.7
|
||||
- Problem: Ignores **variation** in data.
|
||||
|
||||
*** d. Median + Quartiles
|
||||
|
||||
- 1st and 3rd quartiles provide information about data spread.
|
||||
- Still doesn't guarantee ability to distinguish between groups.
|
||||
|
||||
*** e. Statistical Hypothesis Testing
|
||||
|
||||
- Scientific method to determine if observed differences are **statistically significant**.
|
||||
- Necessary for robust and credible comparison of algorithms.
|
||||
|
||||
** 5. Statistical Hypothesis Testing: Process
|
||||
|
||||
1. Define what to compare (e.g., accuracy or fitness).
|
||||
2. Ensure fair comparison:
|
||||
- Equal number of evaluations or explain why not.
|
||||
- Example: Adjust generations to equate computational budget across algorithms.
|
||||
3. Formulate hypotheses:
|
||||
- **Null hypothesis (H₀)**: No difference between the two groups.
|
||||
- **Alternative hypothesis (H₁)**: A statistically significant difference exists.
|
||||
4. Select an appropriate test based on data distribution.
|
||||
|
||||
** 6. Choosing the Test
|
||||
|
||||
*** a. Normality Assumption
|
||||
|
||||
- Many statistical tests assume a **normal distribution** of values.
|
||||
- Visual inspection or tests (e.g., Shapiro-Wilk) can check this.
|
||||
|
||||
*** b. Parametric vs Non-parametric Tests
|
||||
|
||||
- Parametric tests (e.g., t-test):
|
||||
- More powerful.
|
||||
- Require assumptions (e.g., normality, homogeneity of variance).
|
||||
- Non-parametric tests (e.g., Wilcoxon, Mann-Whitney):
|
||||
- Safer for non-normal data.
|
||||
- Widely used in stochastic algorithm comparisons.
|
||||
|
||||
*** c. Paired vs Unpaired Tests
|
||||
|
||||
- **Paired**: Use when comparing results from same initial conditions.
|
||||
- **Unpaired**: Use when runs are completely independent.
|
||||
|
||||
** 7. Test Outputs
|
||||
|
||||
- Test produces a **statistic** and a **p-value**.
|
||||
- If **p ≤ 0.05**, reject H₀: significant difference exists.
|
||||
- If **p > 0.05**, do not reject H₀: no significant difference found.
|
||||
- Significance level is usually set to **0.05**, corresponding to 95% confidence.
|
||||
- Lower significance (e.g., 0.01) may be used in critical applications.
|
||||
|
||||
** 8. Interpreting P-Values
|
||||
|
||||
- High p-value → Observed difference likely due to chance → **Do not reject H₀**.
|
||||
- Low p-value → Observed difference unlikely due to chance → **Reject H₀**.
|
||||
|
||||
** 9. Test Examples (in R)
|
||||
|
||||
- Two-tailed Wilcoxon Rank-Sum Test (unpaired).
|
||||
- Two-tailed Wilcoxon Signed-Rank Test (paired).
|
||||
|
||||
** 10. Multiple Comparisons Problem
|
||||
|
||||
- Comparing many algorithms or configurations increases the risk of **Type I errors** (false positives).
|
||||
- Correction methods:
|
||||
- Adjust the significance threshold (e.g., Bonferroni correction).
|
||||
- Downside: Conservative → **reduced power** (risk of missing real differences).
|
||||
|
||||
** 11. Tests for N Groups
|
||||
|
||||
- Stronger than multiple pairwise tests with correction.
|
||||
- Common tests:
|
||||
- **Kruskal-Wallis Test**: for unpaired comparisons across groups.
|
||||
- **Friedman Test**: for paired comparisons across groups.
|
||||
|
||||
*** Post-hoc Analysis
|
||||
|
||||
- Needed when the global test finds significant differences but doesn't specify **which pairs** differ.
|
||||
- Kruskal-Wallis → Dunn post-hoc test.
|
||||
- Friedman → Nemenyi post-hoc test.
|
||||
|
||||
213
Uni/20250403120140-fyp_report_planning.org
Normal file
213
Uni/20250403120140-fyp_report_planning.org
Normal file
@@ -0,0 +1,213 @@
|
||||
:PROPERTIES:
|
||||
:ID: 26b2ed9a-cb81-4c43-bc63-6b3c8ffa3bf1
|
||||
:END:
|
||||
#+title: fyp-report-planning
|
||||
#+filetags: :uni:
|
||||
|
||||
1. Title Page
|
||||
|
||||
Title: "AI-Assisted Note-Taking Web Application"
|
||||
|
||||
Your Name
|
||||
|
||||
Supervisor's Name
|
||||
|
||||
Institution/Department
|
||||
|
||||
Date
|
||||
|
||||
|
||||
|
||||
2. Abstract
|
||||
|
||||
A concise summary of the project (150–250 words).
|
||||
|
||||
Highlight the problem, solution, methods, and results.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
3. Table of Contents
|
||||
|
||||
Include all headings and subheadings with page numbers.
|
||||
|
||||
|
||||
|
||||
4. Introduction
|
||||
|
||||
4.1 Background: Explain the context of note-taking and its challenges.
|
||||
|
||||
4.2 Problem Statement: Describe the specific problem you aim to solve (e.g., information overload, accessibility).
|
||||
|
||||
4.3 Objectives: Define the goals of your project.
|
||||
|
||||
4.4 Scope: Clearly state the boundaries of the project.
|
||||
|
||||
4.5 Dissertation Structure: Briefly outline the contents of each section.
|
||||
|
||||
|
||||
|
||||
5. Literature Review
|
||||
|
||||
5.1 Existing Solutions: Review existing tools for note-taking, their advantages, and limitations.
|
||||
|
||||
5.2 Related Research: Explore research on AI, NLP, and note-taking technologies.
|
||||
|
||||
5.3 Gaps in the Literature: Identify areas not addressed by current solutions, justifying the need for your project.
|
||||
|
||||
|
||||
|
||||
6. Methodology
|
||||
|
||||
6.1 Problem Analysis: Define the user requirements and personas.
|
||||
|
||||
6.2 Proposed Solution: Describe your AI-assisted note-taking solution conceptually.
|
||||
|
||||
6.3 Technology Stack: Outline the tools, frameworks, APIs, and databases you plan to use.
|
||||
|
||||
6.4 System Architecture: Include diagrams for the client-server architecture and system workflow.
|
||||
|
||||
6.5 Data Handling: Describe how the AI will process input data (e.g., text, audio) and produce outputs.
|
||||
|
||||
|
||||
|
||||
7. Implementation
|
||||
|
||||
7.1 Development Process: Document how you built the system (e.g., Agile methodology).
|
||||
|
||||
7.2 Features: Detail key features, such as AI-powered summarization, search functionality, or collaboration tools.
|
||||
|
||||
7.3 Challenges: Discuss technical challenges and how you addressed them.
|
||||
|
||||
|
||||
|
||||
8. Evaluation
|
||||
|
||||
8.1 Testing: Describe how you tested the application (e.g., user testing, performance metrics).
|
||||
|
||||
8.2 Results: Present quantitative and qualitative results, including user feedback and performance benchmarks.
|
||||
|
||||
8.3 Analysis: Critically analyze the results and their implications.
|
||||
|
||||
|
||||
|
||||
9. Discussion
|
||||
|
||||
9.1 Contributions: Highlight the unique aspects of your solution.
|
||||
|
||||
9.2 Limitations: Address any shortcomings in your system.
|
||||
|
||||
9.3 Future Work: Suggest possible enhancements and future research directions.
|
||||
|
||||
|
||||
|
||||
10. Conclusion
|
||||
|
||||
Summarize the problem, solution, and key findings.
|
||||
|
||||
Reiterate the impact of your work and its importance.
|
||||
|
||||
|
||||
|
||||
11. References
|
||||
|
||||
List all the sources cited in the document using a consistent citation style (e.g., APA, IEEE, Harvard).
|
||||
|
||||
|
||||
|
||||
12. Appendices
|
||||
|
||||
Include any additional materials, such as:
|
||||
|
||||
Code snippets
|
||||
|
||||
User manuals
|
||||
|
||||
Detailed testing data
|
||||
|
||||
Wireframes or UI designs
|
||||
|
||||
* intro
|
||||
# Introduction
|
||||
|
||||
## 1.1 Background
|
||||
|
||||
The digital era has revolutionized the way people consume and manage information. With the proliferation of online content, academic resources, and professional documentation, individuals are constantly processing vast amounts of data. Note-taking, a fundamental cognitive tool for organizing knowledge, has evolved from traditional pen-and-paper methods to digital platforms that offer increased accessibility, storage, and retrieval capabilities. Despite these advancements, users still face challenges such as information overload, inefficient retrieval, and lack of contextual understanding.
|
||||
|
||||
Artificial Intelligence (AI) and Natural Language Processing (NLP) have emerged as transformative technologies in optimizing information management. AI-powered note-taking applications can enhance the process by automatically summarizing content, organizing notes, and enabling intelligent search functionalities. By leveraging machine learning techniques, these systems can provide a personalized and efficient approach to note-taking, reducing cognitive load and improving productivity.
|
||||
|
||||
## 1.2 Problem Statement
|
||||
|
||||
While digital note-taking applications exist, most rely on manual input and basic text organization without leveraging AI to enhance usability. Users often struggle with the sheer volume of notes, leading to difficulties in retrieving relevant information quickly. Traditional search mechanisms lack semantic understanding, making it challenging to locate specific insights. Additionally, manually summarizing large amounts of text is time-consuming and inefficient. There is a growing need for a system that can intelligently process, categorize, and retrieve notes in an intuitive manner.
|
||||
|
||||
## 1.3 Objectives
|
||||
|
||||
This project aims to develop an AI-assisted note-taking web application that enhances knowledge management through NLP and machine learning. The primary objectives include:
|
||||
|
||||
- **Automated Summarization**: Implement AI-powered summarization to extract key points from lengthy notes.
|
||||
- **Intelligent Search**: Develop a smart search function that understands context and retrieves relevant information efficiently.
|
||||
- **Automated Tagging and Organization**: Utilize AI to categorize and label notes for easy navigation.
|
||||
- **Speech-to-Text Integration**: Enable users to convert spoken content into structured notes.
|
||||
- **User-Friendly Interface**: Design an intuitive UI/UX for seamless interaction and accessibility.
|
||||
|
||||
## 1.4 Scope
|
||||
|
||||
This project focuses on developing a web-based AI-assisted note-taking application with core functionalities such as summarization, intelligent search, and speech-to-text conversion. The system will leverage NLP models and machine learning techniques to process textual data effectively. However, the project does not aim to develop a fully autonomous AI capable of understanding handwritten notes or processing multimedia inputs such as images and videos. Future enhancements may incorporate these aspects.
|
||||
|
||||
## 1.5 Dissertation Structure
|
||||
|
||||
This dissertation is structured as follows:
|
||||
|
||||
- **Chapter 2: Literature Review** – Discusses existing note-taking solutions, AI applications in knowledge management, and identifies gaps in the literature.
|
||||
- **Chapter 3: Methodology** – Outlines the technical approach, system architecture, and technologies used in developing the application.
|
||||
- **Chapter 4: Implementation** – Details the development process, features, and challenges encountered.
|
||||
- **Chapter 5: Evaluation** – Presents testing methodologies, results, and user feedback.
|
||||
- **Chapter 6: Discussion** – Analyzes contributions, limitations, and potential future work.
|
||||
- **Chapter 7: Conclusion** – Summarizes findings and the impact of the research.
|
||||
|
||||
By addressing the challenges of digital note-taking with AI, this project aims to provide an innovative solution that enhances productivity and information management.
|
||||
|
||||
* intor 2
|
||||
Absolutely! Here's a draft of the **Introduction** section for your dissertation titled **"AI-Assisted Note-Taking Web Application"**. This version reflects a strong academic tone and addresses all required subheadings (4.1–4.5). Feel free to tweak any specific names or phrases to better match your own work:
|
||||
|
||||
---
|
||||
|
||||
## 4. Introduction
|
||||
|
||||
### 4.1 Background
|
||||
|
||||
Note-taking plays a fundamental role in both academic and professional contexts, serving as a critical tool for learning, information retention, and productivity. Traditional methods of note-taking, whether handwritten or digital, often rely heavily on user input, which can be cognitively demanding, time-consuming, and inconsistent across users. In the age of digital information, individuals are increasingly overwhelmed by the volume and complexity of content encountered during lectures, meetings, or research. This has led to growing concerns about cognitive overload, especially when users must process, organize, and retrieve large quantities of data manually.
|
||||
|
||||
With the advancement of Artificial Intelligence (AI) and Natural Language Processing (NLP), there is a growing potential to revolutionize the way notes are captured, organized, and utilized. By leveraging machine learning algorithms, semantic understanding, and contextual summarization, AI can assist users in real-time or post-session to generate coherent and concise notes. However, the integration of such technologies into practical, user-friendly applications remains an ongoing challenge.
|
||||
|
||||
### 4.2 Problem Statement
|
||||
|
||||
Despite the availability of numerous note-taking tools, many fail to address critical user pain points such as cognitive overload, poor summarization, lack of personalization, and limited accessibility. Students, professionals, and individuals with neurodivergent needs may find it especially difficult to engage with existing solutions that require constant manual interaction. There remains a significant gap in tools that can intelligently process raw input (e.g., text or speech), summarize key points, and provide structured, searchable output that adapts to the user’s workflow. This project aims to bridge that gap by developing an AI-assisted web application that automates and enhances the note-taking process.
|
||||
|
||||
### 4.3 Objectives
|
||||
|
||||
The primary objectives of this dissertation project are:
|
||||
|
||||
- To design and implement a web-based note-taking application that integrates AI capabilities for summarization and organization.
|
||||
- To explore the use of NLP and machine learning techniques for processing text-based inputs.
|
||||
- To improve accessibility and reduce cognitive load for users by automating key aspects of the note-taking process.
|
||||
- To evaluate the effectiveness and usability of the system through user feedback and performance metrics.
|
||||
|
||||
### 4.4 Scope
|
||||
|
||||
This project focuses on the development of a prototype web application tailored to textual note input, though the underlying architecture may be extended to support audio input in future iterations. The system will include core features such as AI-powered summarization, keyword extraction, and searchable note archives. It will be built using Angular for the frontend, Spring Boot for the backend, and PostgreSQL for data storage. OpenAI’s API will be utilized to integrate advanced language processing capabilities. While the project does not aim to develop novel NLP models, it will focus on the integration and optimization of existing tools to deliver a practical, user-centric solution.
|
||||
|
||||
### 4.5 Dissertation Structure
|
||||
|
||||
This dissertation is structured as follows:
|
||||
- **Chapter 5: Literature Review** examines current note-taking tools, relevant AI technologies, and identifies gaps that justify this project.
|
||||
- **Chapter 6: Methodology** describes the user research, system design, technology stack, and system architecture.
|
||||
- **Chapter 7: Implementation** details the development process, key features, and technical challenges faced during the build.
|
||||
- **Chapter 8: Evaluation** outlines the testing methods used and presents results with analysis.
|
||||
- **Chapter 9: Discussion** reflects on the project’s contributions, limitations, and potential directions for future work.
|
||||
- **Chapter 10: Conclusion** summarizes the project and reiterates its significance.
|
||||
|
||||
---
|
||||
|
||||
Would you like help tailoring this to reflect specific design decisions you've already made, like how your AI summarizer works or your user testing process?
|
||||
35
Uni/20250723184233-advanced_networking.org
Normal file
35
Uni/20250723184233-advanced_networking.org
Normal file
@@ -0,0 +1,35 @@
|
||||
:PROPERTIES:
|
||||
:ID: 3acffb66-bc1a-4661-904f-c5447b3c3488
|
||||
:END:
|
||||
#+title: Advanced Networking
|
||||
#+filetags: :uni:networking:
|
||||
|
||||
# Advanced Networking
|
||||
|
||||
This module was taught in the first semester of my final year at university. There were a wide range of topics that were covered, and best efforts were made in not talking about the security aspects of networks, although it is the case that when we talk about networks we are talking about secure networks (who even needs insecure networks?).
|
||||
|
||||
The following core topics were covered:
|
||||
|
||||
1. Lower Layer Protocols
|
||||
|
||||
- Packet vs Circuit Switching, Ethernet, layer models (DoD 4/5, OSI 7).
|
||||
- Network Hardware: Switches, Routers, data/control/management plane. Software defined networks.
|
||||
- LAN/WAN split, Arpanet, DoD, OSI. Why OSI Failed
|
||||
- Link aggregation and VLANs
|
||||
|
||||
2. IP Addressing
|
||||
|
||||
- Addressing, routing, concepts. Why IPv6 is needed
|
||||
- Address allocation, bootp, DHCP, SLAAC
|
||||
- NAT and Proxying
|
||||
|
||||
3. TCP/UDP
|
||||
- UDP: applications, advantages and disadvantages
|
||||
- TCP: applications, advantages and disadvantages, mechanisms and operation, sequence numbers, receive windows, slow start, window scaling, PAWS, timestamping, multipathing
|
||||
- Demultiplexing, multiplexing
|
||||
|
||||
4. DNS
|
||||
- DNS: concepts, resource records, RR sets, basic operation, recursive and authoritative servers, caching, DNSSEC
|
||||
|
||||
5. Higher layer protocols
|
||||
- Basic operation of HTTP, FTP, SMTP
|
||||
Reference in New Issue
Block a user