2.2 KiB
Executable File
2.2 KiB
Executable File
ise_week_1
Bug report classification
Perform bug report classification with AI following the steps below:
- Text filtering
- Stemming
- Indexing
- Train machine learning models, i.e., Decision Tree, Naive Bayes, and Logistic Regression
- Prediction (after data pre-processing)
Text filtering:
- Punctuation removal
- Specific transformations occur: the myUser has no meaning, its just a word. You need to split it into two meaningful words.
- Stop words should not be removed, as it changes the whole meaning of a bug report.
- Unlike other NLP tasks, removing stop words may distort the meaning of bug reports.
Stemming:
- Stemming reduces different forms of a word to its common base by sequential application of stemming rules. For example: light caresses colours becomes Light caress colour
Indexing:
- Indexing: we need to convert the text into numeric representation, there are two common ways of doing this: TF and IDF
- TF: total words / specific word . for example: "bug" appears 50 times in a 150 terms.
- IDF: We penalise certain words based on their frequency.
- bug report classifications here often does not use IDF but TF, because some frequent terms should not be penalized
- For each report, we will have a vector of indexing for words in the reports according to a dictionary
- Example: A bug report = {“this”, “bugs”, “failure”, “interesting”, …} Vector = <0.123, 0.34, 0.1, ….>
DT
- What is decision tree: a tree structure that split the data depending on different values.
- Each node is a test on the attribute.
- Each branch represents the outcome.
NB
- What is Naive Bayes? Classifier based on the conditional probability given by the Bayes theorem.
- Calculating the probability of each class, and classify the given features into the one with higher probability.
LR
- What is logistic regression: a linear model for classification
- Fitting the logistic growth and the sigmoid midpoint
- Output a probability, but can use a cutoff point to decide class.
Results:
Choose the top 20/50 words when theyre used as a feature