57 lines
2.2 KiB
Org Mode
57 lines
2.2 KiB
Org Mode
:PROPERTIES:
|
|
:ID: 06b2a012-4e8a-4a8a-9494-de7ed9fbe1d3
|
|
:END:
|
|
#+title: ise_week_1
|
|
#+filetags: :uni:notes:
|
|
|
|
|
|
* Bug report classification
|
|
|
|
Perform bug report classification with AI following the steps
|
|
below:
|
|
|
|
- Text filtering
|
|
- Stemming
|
|
- Indexing
|
|
- Train machine learning models, i.e., Decision Tree, Naive Bayes, and Logistic
|
|
Regression
|
|
- Prediction (after data pre-processing)
|
|
|
|
** Text filtering:
|
|
- Punctuation removal
|
|
- Specific transformations occur:
|
|
the myUser has no meaning, its just a word. You need to split it into two meaningful words.
|
|
- Stop words should not be removed, as it changes the whole meaning of a bug report.
|
|
- Unlike other NLP tasks, removing stop words may distort the meaning of bug reports.
|
|
|
|
** Stemming:
|
|
- Stemming reduces different forms of a word to its common base by sequential application of stemming rules. For example: light caresses colours becomes Light caress colour
|
|
|
|
** Indexing:
|
|
|
|
- Indexing: we need to convert the text into numeric representation, there are two common ways of doing this: TF and IDF
|
|
- TF: total words / specific word . for example: "bug" appears 50 times in a 150 terms.
|
|
- IDF: We penalise certain words based on their frequency.
|
|
- bug report classifications here often does not use IDF but TF, because some frequent terms should not be penalized
|
|
|
|
- For each report, we will have a vector of indexing for words in the reports according to a dictionary
|
|
- Example: A bug report = {“this”, “bugs”, “failure”, “interesting”, …}
|
|
Vector = <0.123, 0.34, 0.1, ….>
|
|
|
|
** DT
|
|
- What is decision tree: a tree structure that split the data depending on different values.
|
|
- Each node is a test on the attribute.
|
|
- Each branch represents the outcome.
|
|
|
|
** NB
|
|
- What is Naive Bayes? Classifier based on the conditional probability given by the Bayes theorem.
|
|
- Calculating the probability of each class, and classify the given features into the one with higher probability.
|
|
|
|
** LR
|
|
- What is logistic regression: a linear model for classification
|
|
- Fitting the logistic growth and the sigmoid midpoint
|
|
- Output a probability, but can use a cutoff point to decide class.
|
|
|
|
** Results:
|
|
Choose the top 20/50 words when theyre used as a feature
|