Where we last left off in Part 1 of Diving deeper into Bible NLP Visualization, we used Python and created a dataframe of the King James bible text file that looked like so, with books, chapters, verse numbers, and verse text separated into their own columns:

The dataframe is now a helpful tool for efficient information retrieval. We can ask questions such as “how many verses are contained in Genesis?”, by referencing appropriate Python filters and groupby statements with correspondingly appropriate aggregations. However, we need to do more work on the dataframe in order to visualize semantic meaning effectively. We next need to generate a semantic space.

Bible Semantic Space to enable Bible NLP

We feed the text of the verse column into a word embedding model to generate a semantic space. Word embedding models are tools in natural language processing to translate words and/or the context of words into vector representations so we can direct our computer to a mathematical object to analyze. Here’s an analogy to explain the concept. In Einsteinian physics, position is not determined by an absolute space time grid, position is relative to the position of other objects and their movement.

Word embedding models function somewhat similarly. Word embedding models are initially trained on massive corpus’/bodies of linguistic data. Corpus’ are datasets of combined sources such as Wikipedia, the set of all tweets, an agglomeration of the last 20 years’ worth of several publishers ‘ news articles, etc. From this set of information meaning is extracted by way of cooccurrence of words and phrases.

Word2vec

We have chosen word2vec as the embedding model for our initial analysis within the gensim package. Word2vec was created in 2013 by a team of Google researchers, including Tomas Mikolov.

Depending upon our choice of semantic embedding model, we must utilize different preprocessing techniques to get our dataframe ready. We will use gensim to preprocess our strings into a format most advantageous for word2vec. Preprocessing consists of a number of techniques for manipulating your text for better machine representation, such as:

• Removal of punctuation
• Removal of stop words
• Change case to lower
• Stemming
• Lemmatizing
• Tokenization

There is room for choice here, for example our selection of stopwords to eliminate will alter the semantic space generated. Typical stopwords eliminated include (‘and’,’or’,’the’,’a’,etc.). It is also possible with modern embedding models to include all stopwords in the corpus. Each word is assigned a numerical token representation like below:

Lastly, each tokenized word is assigned a point in a N-dimensional space. One can choose the number of dimensions they wish to use. The general thought is that more dimensions are able to capture more sophisticated linguistic meaning. We will be using an N of 100 for our initial runs with word2vec. Several modern models extend their count of parameters into the billions to achieve state of the art performance. We can see the model output below.

We can use dimension reduction on the above object to reduce it to 2 dimensions for interpretable plotting. We will cover that and more in a future post.

The post Generating a Bible Semantic Space appeared first on Better Biblos.

Introduction to Transforming the Bible

In this first post of a series of posts about our Bible NLP Analysis, we will be transforming the Bible into a dataframe. Part 1 will focus on the ingestion of the file into a broadly usable Pandas dataframe.

I utilized a .txt file of the King James Bible from the website sacred-texts.com. Their formatting of the file enabled for much easier ingestion of the text in a logical way than other sources I have found.

Python & Bible NLP

My language of choice for NLP is Python. Python has many excellent modules to aid in various object-oriented tasks, NLP included. Pandas is one such module in the data science realm. One can find links to many of the resources I utilize here on the Better Biblos resources page. In addition to many other functions, Pandas allows for easy ingestion of the .txt file using:

!pip install pandas
import pandas as pd
bible = pd.read_csv(r'...\kjvdat.txt')
bible.head()

Inspecting the object using bible.head() we observe the following tabular structure:

Not too bad, but we can utilize the logical structure of the Bible into books, chapters, and verses to create a more accessible object. In particular we will focus on the pipe (“|”) delimiter character nicely included in the file, and then force name the automatically generated columns from the .split command in line 1 below:

bible = bible[0].str.split('|', n=-1, expand=True)
bible.columns = ['book','chapter','verse_number','verse']
bible.head()

We can now very easily query subsets of data by books, chapters, and verses.

The post Transforming the Bible into a Dataframe appeared first on Better Biblos.