Boardsload274

Now, we need to tokenize the sentences into words aka terms. While we can do it in a loop, we can take advantage of the split function in the text toolkit for Pandas’ Series; see this manual for all the functions. To get it we just invoke the strip function, which is a part of str, i.e. StringsMethods object. The argument is regular.

• Tokenize Pandas Column Definition
• Tokenize Pandas Columns
• Tokenize Pandas Column Names
• Tokenize Pandas Column Tutorial

• Pandas Pandas: An on-the-go “cheat sheet” PRO TIP: do a ctrl f first python - How to select rows from a DataFrame based on column values - Stack Overflow.
• How to apply postagsents to pandas dataframe efficiently. How to use wordtokenize in data frame. How to apply postagsents to pandas dataframe efficiently. Tokenizing words into a new column in a pandas dataframe. Run nltk senttokenize through Pandas dataframe. Python text processing: NLTK and pandas.
• Edit: You could be thinking the Dataframe df after series.apply(nltk.wordtokenize) is larger in size, which might affect the runtime for the next operation dataframe.apply(nltk.wordtokenize). Pandas optimizes under the hood for such a scenario. I got a similar runtime of 200s by only performing dataframe.apply(nltk.wordtokenize) separately.

Posted By Jakub Nowacki, 18 September 2017

Pandas is a great tool for the analysis of tabulardata via its DataFrame interface. Slightly less known are its capabilities forworking with text data. In this post I’ll present them on some simple examples.As a comparison I’ll use my previous post about TF-IDF in Spark.

First we load data, i.e. books previously downloaded from Project Gutenbergsite. Here we use a handy globmodule to walk over text files ina directory and read in files line by line into DataFrame.

Since we’re now in Pandas, we can easily use its plottingcapability tolook at the number of lines in the books. Note that value_counts of bookscreate Series indexed by the book names, thus, we don’t need to set index forplotting. Note that we can now choose plot styling; see demopagefor available styles.

Now, we need to tokenize the sentences into words aka terms. While we can do itin a loop, we can take advantage of the split function in the text toolkit forPandas’ Series; see this manual for all the functions. To get it we just invoke thestrip function, which is a part of str, i.e. StringsMethods object. Theargument is regular expression pattern, on which the string, in our case thesentence, will be split. In our case it is the regular expression set ofeverything that is not a letter (capital or not) or digit and underscore. Thisis because W is the reverse of w which is a set [A-Za-z0-9_], i.e.contains an underscore. In this case we want also to split on underscore so wehad to add it to the splitting set. Note that this decision and other you’llmake while deciding on the splitting set, will influence the tokenization, andthus the result. Hence, select the splitting pattern carefully. If you’re notsure about your decision, use practicing pages, likeregex101.com to check your patterns.

As a result we get a new column with array of words. Now we need to flatten theresulted DataFrame somehow. The best approach is to use iterators and create newDataFrame with words; see this StackOverflowpost for the performance details of this solution. Note that thismay take a few seconds depending on the machine; on mine it takes about 25s.

Now we have DataFrame with words, but occasionally we get an empty string as abyproduct of the splitting. We should remove it as it would be counted as aterm. To do that we can use function len as follows:

The newwords DataFrame is now free of the empty strings.

If we want to calculate TF-IDF statistic we need to normalize the words bybringing them all to the same case. Again, we can use a Series string functionfor that.

Tokenize Pandas Column Definition

First, lets calculate counts of the terms per book. We can do it as follows:

Note that as a result we are getting a hierarchical index aka MultiIndex; seePandas manual formore details.

With this index we can now plot the results in much nicer way. E.e. we can get 5words largest counts per book and plot it as shown below. Note that we need toreset, i.e. remove one level of index, as it doubles when we are usingnlargest function. I’ve made the process into a function, as I will reuse itfurther.

To finish the TF as shown in the previous post, we need the counts of the books to get the measure. As a reminder, below is the equation for TF:

Note that I’ve renamed the column as it inherits the name by default.

Now we need to join both columns on book to get the sum of word per book intofinal DataFrame. Having both n_w and n_d, we can easily calculate TF asfollows:

Again, lets look at the top 5 words with respect to TF.

As before (and as expected), we’ve got the English stop-words.

Now we can do IDF; the remainder of the formula is given below:

First we need to get the document count. The simples solution is to use theSeries’ method nunique, i.e. get size of set of unique elements in a series.

Similarly, to get the number of unique books that every term appeared in, we canuse the same method but on a grouped data. Again we need to rename the column.Note that sorting values is only for the presentation and it is not needed forthe further computations.

Having all the components of the IDF formula, we can now calculate it as a newcolumn.

To get the final DataFrame we need to join both DataFrames as follows:

Tokenize Pandas Columns

Having now DataFrame with both TF and IDF values, we can calculate TF-IDFstatistic.

Again, lets see the top TF-IDF terms:

Tokenize Pandas Column Names

As before we’ve got a set of important words for the given document.

Note that this more of a demo of Pandas text processing. If you’re consideringusing TF-IDF in a more production example, see some existing solutions likescikit-learn’s TfidfVectorizer.

Note that I’ve just scratched a surface with the Pandas’ text processingcapabilietes. There are many other useful functions like the match functionshown below:

Tokenize Pandas Column Tutorial

Again, I encourage you to check Pandas manual for more details.

Notebook with the above computations is available for downloadhere.