Demo: Creating and inspecting a narrative graph¶
This notebook will serve as a demo and small tour of some of the core functionalities of a NarrativeGraph object.
Data setup¶
For this demo notebook, we will be using News Category Dataset [1, 2] available on Kagglehub because it has short texts, timestamps and are categorized.
In [1]:
Copied!
import time
from kagglehub import KaggleDatasetAdapter
import kagglehub
data = kagglehub.dataset_load(
KaggleDatasetAdapter.PANDAS,
"rmisra/news-category-dataset",
"News_Category_Dataset_v3.json",
pandas_kwargs=dict(lines=True),
)
data.head()
import time
from kagglehub import KaggleDatasetAdapter
import kagglehub
data = kagglehub.dataset_load(
KaggleDatasetAdapter.PANDAS,
"rmisra/news-category-dataset",
"News_Category_Dataset_v3.json",
pandas_kwargs=dict(lines=True),
)
data.head()
/Users/au479461/PycharmProjects/narrative-graph/.venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
Out[1]:
| link | headline | category | short_description | authors | date | |
|---|---|---|---|---|---|---|
| 0 | https://www.huffpost.com/entry/covid-boosters-... | Over 4 Million Americans Roll Up Sleeves For O... | U.S. NEWS | Health experts said it is too early to predict... | Carla K. Johnson, AP | 2022-09-23 |
| 1 | https://www.huffpost.com/entry/american-airlin... | American Airlines Flyer Charged, Banned For Li... | U.S. NEWS | He was subdued by passengers and crew when he ... | Mary Papenfuss | 2022-09-23 |
| 2 | https://www.huffpost.com/entry/funniest-tweets... | 23 Of The Funniest Tweets About Cats And Dogs ... | COMEDY | "Until you have a dog you don't understand wha... | Elyse Wanshel | 2022-09-23 |
| 3 | https://www.huffpost.com/entry/funniest-parent... | The Funniest Tweets From Parents This Week (Se... | PARENTING | "Accidentally put grown-up toothpaste on my to... | Caroline Bologna | 2022-09-23 |
| 4 | https://www.huffpost.com/entry/amy-cooper-lose... | Woman Who Called Cops On Black Bird-Watcher Lo... | U.S. NEWS | Amy Cooper accused investment firm Franklin Te... | Nina Golgowski | 2022-09-22 |
The columns that we will be using as input for our narrative graph.
- Documents: headline + short_description
- IDs: link, but without the part that is in all of them
- Timestamps: date
- Categories: category
There are many categories. We will create a subset with just two of them: U.S. News and Politics.
In [2]:
Copied!
# create a sample
sample = data[data["category"].isin(["U.S. NEWS", "POLITICS"])].sample(
5000, random_state=42
)
docs = sample["headline"] + "\n\n" + sample["short_description"]
ids = sample["link"].replace("https://www.huffpost.com/entry/", "") # get rit of the first part of the URL
categories = sample["category"]
timestamps = sample["date"]
# create a sample
sample = data[data["category"].isin(["U.S. NEWS", "POLITICS"])].sample(
5000, random_state=42
)
docs = sample["headline"] + "\n\n" + sample["short_description"]
ids = sample["link"].replace("https://www.huffpost.com/entry/", "") # get rit of the first part of the URL
categories = sample["category"]
timestamps = sample["date"]
Creating the model¶
Once we have our list of documents, which is the only required input, and extra metadata in aligned lists, we can create a narrative graph.
In [3]:
Copied!
from narrativegraphs import NarrativeGraph
model = NarrativeGraph()
model.fit(docs, doc_ids=ids, categories=categories, timestamps=timestamps)
from narrativegraphs import NarrativeGraph
model = NarrativeGraph()
model.fit(docs, doc_ids=ids, categories=categories, timestamps=timestamps)
Extracting triplets: 0%| | 0/5000 [00:00<?, ?it/s]
Extracting triplets: 0%| | 1/5000 [00:01<2:19:28, 1.67s/it]
Extracting triplets: 3%|████████▏ | 169/5000 [00:01<00:36, 132.56it/s]
Extracting triplets: 7%|██████████████████▏ | 372/5000 [00:01<00:14, 323.99it/s]
Extracting triplets: 10%|█████████████████████████▎ | 518/5000 [00:03<00:28, 157.41it/s]
Extracting triplets: 15%|███████████████████████████████████▊ | 734/5000 [00:03<00:15, 271.90it/s]
Extracting triplets: 19%|██████████████████████████████████████████████▌ | 955/5000 [00:03<00:09, 418.07it/s]
Extracting triplets: 22%|██████████████████████████████████████████████████████▍ | 1120/5000 [00:05<00:17, 216.76it/s]
Extracting triplets: 27%|█████████████████████████████████████████████████████████████████▎ | 1344/5000 [00:05<00:11, 324.02it/s]
Extracting triplets: 30%|████████████████████████████████████████████████████████████████████████▉ | 1501/5000 [00:07<00:17, 203.45it/s]
Extracting triplets: 34%|███████████████████████████████████████████████████████████████████████████████████▊ | 1724/5000 [00:07<00:10, 299.38it/s]
Extracting triplets: 39%|██████████████████████████████████████████████████████████████████████████████████████████████▉ | 1953/5000 [00:07<00:07, 425.51it/s]
Extracting triplets: 42%|███████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 2122/5000 [00:08<00:12, 237.53it/s]
Extracting triplets: 47%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 2343/5000 [00:08<00:07, 336.94it/s]
Extracting triplets: 50%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 2501/5000 [00:10<00:11, 217.06it/s]
Extracting triplets: 55%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 2730/5000 [00:10<00:07, 314.86it/s]
Extracting triplets: 59%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 2946/5000 [00:10<00:04, 431.25it/s]
Extracting triplets: 62%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 3112/5000 [00:12<00:07, 241.53it/s]
Extracting triplets: 67%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 3329/5000 [00:12<00:04, 339.52it/s]
Extracting triplets: 70%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 3501/5000 [00:13<00:06, 219.13it/s]
Extracting triplets: 74%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 3721/5000 [00:13<00:04, 311.51it/s]
Extracting triplets: 79%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 3949/5000 [00:13<00:02, 434.43it/s]
Extracting triplets: 82%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 4116/5000 [00:15<00:03, 250.83it/s]
Extracting triplets: 87%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 4358/5000 [00:15<00:01, 363.47it/s]
Extracting triplets: 90%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 4515/5000 [00:17<00:02, 223.18it/s]
Extracting triplets: 95%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 4740/5000 [00:17<00:00, 318.30it/s]
Extracting triplets: 99%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 4973/5000 [00:17<00:00, 444.72it/s]
Extracting triplets: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5000/5000 [00:17<00:00, 288.24it/s]
Out[3]:
<narrativegraphs.graphs.NarrativeGraph at 0x119c06ff0>
Inspecting the model visually¶
One of the key features of the narrativegraphs package is that it lets a user inspect the output interactively in a browser-based visualizer. It is hosted directly on your machine by the Python package – no extra dependencies required. This is achieved with the one line below.
Click the link in the log messages to open in your browser.
In [4]:
Copied!
# create server to be viewed in own browser which blocks execution of other cells
## model.serve_visualizer()
## Or run in the background
server = model.serve_visualizer(block=False)
# create server to be viewed in own browser which blocks execution of other cells
## model.serve_visualizer()
## Or run in the background
server = model.serve_visualizer(block=False)
In [5]:
Copied!
server.stop()
server.stop()
Stop it by hitting the stop button on the cell in Jupyter Notebook or hit CTRL+C elsewhere.
Inspecting and accessing the model programmatically¶
The graph consists of entities as nodes and their relations or cooccurrences as edges. These, along with the data that back them, like documents and extracted semantic triplets, can be retrieved from the model through properties or service attributes.
Attributes¶
We can get the graph as a whole, as NetworkX graph, through the properties .relation_graph_ and .cooccurrence_graph_.
In [6]:
Copied!
relation_graph = model.relation_graph_
relation_graph = model.relation_graph_
In [7]:
Copied!
print(type(relation_graph))
print(type(relation_graph))
<class 'networkx.classes.digraph.DiGraph'>
In [8]:
Copied!
print(*list(relation_graph.nodes(data=True))[:3], sep="\n")
print(*list(relation_graph.nodes(data=True))[:3], sep="\n")
(1, {'id': 1, 'label': 'Election Day Past', 'frequency': 1, 'focus': False})
(2, {'id': 2, 'label': 'the ghost', 'frequency': 1, 'focus': False})
(3, {'id': 3, 'label': 'Election Day', 'frequency': 4, 'focus': False})
Similarly, entities and relations and everything else can be accessed as pandas.DataFrames through properties.
In [9]:
Copied!
model.entities_
model.entities_
Out[9]:
| id | label | frequency | doc_frequency | adjusted_tf_idf | first_occurrence | last_occurrence | alt_labels | category | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | Election Day Past | 1 | 1 | 0.0 | 2014-11-02 | 2014-11-02 | [] | [POLITICS] |
| 1 | 2 | the ghost | 1 | 1 | 0.0 | 2014-11-02 | 2014-11-02 | [] | [POLITICS] |
| 2 | 3 | Election Day | 4 | 3 | 3750.0 | 2014-11-01 | 2016-09-28 | [] | [POLITICS, POLITICS, POLITICS, POLITICS] |
| 3 | 4 | the early hours | 1 | 1 | 0.0 | 2014-11-02 | 2014-11-02 | [] | [POLITICS] |
| 4 | 5 | An App | 1 | 1 | 0.0 | 2022-03-11 | 2022-03-11 | [] | [POLITICS] |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 7376 | 7377 | Rank-and-file voters | 1 | 1 | 0.0 | 2017-01-19 | 2017-01-19 | [] | [POLITICS] |
| 7377 | 7378 | Russian-aligned Wikileaks | 1 | 1 | 0.0 | 2017-01-04 | 2017-01-04 | [] | [POLITICS] |
| 7378 | 7379 | Jerusalem | 2 | 1 | 2500.0 | 2017-12-16 | 2017-12-16 | [] | [POLITICS, POLITICS] |
| 7379 | 7380 | James M. Dorsey US president Donald J. Trump’s... | 1 | 1 | 0.0 | 2017-12-16 | 2017-12-16 | [] | [POLITICS] |
| 7380 | 7381 | the stage | 1 | 1 | 0.0 | 2017-12-16 | 2017-12-16 | [] | [POLITICS] |
7381 rows × 9 columns
The properties (with trailing _) are nice in that they give back the data in well-known formats that one can continue working with, e.g. NetworkX graphs for graph algorithms and DataFrames for statistical analyses.
Service attributes¶
However, the service attributes offer more control and may be especially handy if the model is quite big, so that you do not necessarily want everything spit out at once.
For instance, you can search for entities with the entities service.
In [10]:
Copied!
democrats_matches = model.entities.search("democrats")
democrats_matches[:10]
democrats_matches = model.entities.search("democrats")
democrats_matches[:10]
Out[10]:
[EntityLabel(id=240, label='Democrats'), EntityLabel(id=5185, label="Democrats' Rebuttal"), EntityLabel(id=3253, label="Democrats' big reform bill"), EntityLabel(id=1202, label='House Democrats'), EntityLabel(id=1968, label='U.S. Democrats'), EntityLabel(id=2492, label='Senate Democrats'), EntityLabel(id=309, label='Two Democrats'), EntityLabel(id=1353, label='2016 Democrats'), EntityLabel(id=398, label='Iowa Democrats'), EntityLabel(id=6415, label='Taunt Democrats')]
In [11]:
Copied!
democrats_id = democrats_matches[0].id
democrats_id = democrats_matches[0].id
And you can create a subgraph that expands from a set of focus nodes and only includes those that pass a filter.
In [12]:
Copied!
from datetime import date
from narrativegraphs import GraphFilter
democrats_graph = model.graph.expand_from_focus_entities(
[democrats_id],
"relation",
graph_filter=GraphFilter(
categories={'category': ["POLITICS"]},
earliest_date=date(2014, 1, 1)
)
)
# stripping labels to remove some whitespaces
print("NODES")
for node in democrats_graph.nodes:
print(node.id, node.label.strip())
print("\nEDGES")
for edge in democrats_graph.edges:
print(edge.subject_label.strip(), '--', edge.label, '->', edge.object_label.strip())
from datetime import date
from narrativegraphs import GraphFilter
democrats_graph = model.graph.expand_from_focus_entities(
[democrats_id],
"relation",
graph_filter=GraphFilter(
categories={'category': ["POLITICS"]},
earliest_date=date(2014, 1, 1)
)
)
# stripping labels to remove some whitespaces
print("NODES")
for node in democrats_graph.nodes:
print(node.id, node.label.strip())
print("\nEDGES")
for edge in democrats_graph.edges:
print(edge.subject_label.strip(), '--', edge.label, '->', edge.object_label.strip())
NODES 240 Democrats 241 Republicans Flip Virginia House 242 a majority 404 Obama 478 Senate 763 A Landslide 891 Trump Voters 907 Bill 1035 More GOP-Held State House Seats 1080 different findings 1081 This committee 1253 good news 1454 Calls 1497 Tehran 1545 judicial nominee Steven Menashi 1573 the Ohio Republican's slam 1633 a bitter partisan battle 1652 This Billionaire Environmental Activist 1967 Key States 2021 Pennsylvania Republican 2112 Republican Lt. Gov. Kim Guadagno 2388 the ballot box 2389 Special Election 2390 Resources 2691 More Clarity 2920 Eyeing 2018 Senate Takeover 3033 Little Time 3072 Inaugural Address Worthy 3176 Hugh Hewitt 3481 The Longest-Serving Woman 3836 Hemp 4052 Nationwide Day 4216 Black candidates 4263 a scapegoat 4394 For Independent Commission 4485 More Than A Dozen State Legislative Chambers 4697 Rep. Ruben Kihuen 4811 Renewed Push 4914 Oversight Committee 5236 Their Nadir 5324 Investigations 5698 Probe 6424 Roughly 40 percent 6430 Short But Outperform Expectations 6711 Biden's Decision 6752 Trump's Voter Fraud Investigation 6918 Suspected Russian Role 7319 their worst electoral position EDGES Democrats -- conceded -> a majority Democrats -- needs -> Trump Voters Democrats -- advanced, Calls -> Bill Democrats -- Flip -> More GOP-Held State House Seats Democrats -- Has -> different findings Democrats -- on -> This committee Democrats -- asked -> judicial nominee Steven Menashi Democrats -- Defeated -> Republican Lt. Gov. Kim Guadagno Democrats -- won -> Special Election Democrats -- dominated -> the ballot box Democrats -- Pumped -> Resources Democrats -- Demands -> More Clarity Democrats -- Begin -> Eyeing 2018 Senate Takeover Democrats -- fled -> Obama Democrats -- led -> Key States Democrats -- Has -> Little Time Democrats -- for -> Senate Democrats -- Became -> The Longest-Serving Woman Democrats -- Agree -> Hemp Democrats -- led -> Nationwide Day Democrats -- nominate -> Black candidates Democrats -- Renew -> Calls Democrats -- cast -> a scapegoat Democrats -- Push -> For Independent Commission Democrats -- won -> More Than A Dozen State Legislative Chambers Democrats -- Calls -> Rep. Ruben Kihuen Democrats -- made -> Renewed Push Democrats -- asked -> Oversight Committee Democrats -- needs -> A Landslide Democrats -- is -> Their Nadir Democrats -- Calls -> Probe Democrats -- Want -> Investigations Democrats -- Comes -> Short But Outperform Expectations Democrats -- Supporting -> Biden's Decision Democrats -- Participating -> Trump's Voter Fraud Investigation Democrats -- of -> Suspected Russian Role Democrats -- is -> their worst electoral position Republicans Flip Virginia House -- to -> Democrats a majority -- of -> Democrats a majority -- in -> Senate good news -- for -> Democrats Tehran -- see -> Democrats the Ohio Republican's slam -- of -> Democrats a bitter partisan battle -- With -> Democrats This Billionaire Environmental Activist -- picked -> Democrats Pennsylvania Republican -- Want -> Democrats Inaugural Address Worthy -- of -> Democrats Hugh Hewitt -- Backs -> Democrats Roughly 40 percent -- of -> Democrats
Saving and loading the model¶
We can save the model for later use, especially if we have a lot of documents that takes a while to process.
In [13]:
Copied!
model.save_to_file("demo.db", overwrite=True)
model.save_to_file("demo.db", overwrite=True)
And we can load it from that saved file.
In [14]:
Copied!
model = NarrativeGraph.load("demo")
model = NarrativeGraph.load("demo")
References¶
[1] Misra, Rishabh. "News Category Dataset." arXiv preprint arXiv:2209.11429 (2022).
[2] Misra, Rishabh and Jigyasa Grover. "Sculpting Data for ML: The first act of Machine Learning." ISBN 9798585463570 (2021).