Getting newspaper article data and saving it to a SQL database

 This week, I decided I would like to learn more about working with text, starting by building some wordclouds based on Newspaper data. At the same time, I wanted to improve my knowledge of collecting data and saving it to a database.

My main goal here was to get and save some interesting data I could work with. I thought newspaper articles for the months of 2020 could be interesting to see the main subjects in the news each month.

Initially, I looked at different newspaper archives out there for articles. There are many great ones, but not many I could find that offer an API. I discovered that the New York Times does have an API with data related to their articles.

This is an article that documents my learning journey, getting the data from the API and saving it to a SQLite


The APIs are accessible at https://developer.nytimes.com/apis

I found the documenation for the APIs (including the FAQ at https://developer.nytimes.com/faq) very good for getting started.

There are a number of different APIs available from the New York Times, including the Books API, which has information on best sellers lists and book reviews, the Film API, for film reviews, and the Community API, which has information on user generated content such as comments.

In this article, I am using the Archive API (https://developer.nytimes.com/docs/archive-product/1/overview). If you pass a year and month to the API, it returns that month’s article data. The article data tells us many things, including where the full article is available on the NYT website, when it was published, its headline and its lead paragraph.

There are three steps I go through here. I:

  1. Connect to the API to explore the data available and see what data I want to save.
  2. Build a database table based on the data I want to save.
  3. Get data for each month of the year and save to the DB.


_______________________________________________________________________________

More info:

Website link: https://databasescientist.org/

Contact us:  contact@databasescientist.org

Nomination Link: https://databasescientist.org/award-nomination/?ecategory=Awards&rcategory=Awardee

_______________________________________________________________________________

social media:

Twitter: https://x.com/databasesc10061

Pinterest: https://in.pinterest.com/databasescientist/

Linked in: https://www.linkedin.com/in/databasescientist-database-440a12365/

You tube: https://www.youtube.com/@databasescientist


Comments

Post a Comment

Popular posts from this blog

Large Language Models and Vector Databases for News Recommendations

Image
  Large Language Models   (LLMs) generated a global buzz in the machine learning community with recent releases of generative AI tools such as Chat-GPT, Bard, and others alike. One of the core ideas behind these solutions is to compute a numerical representation of unstructured data (such as texts and images) and find similarities between these representations. However, taking all of these concepts into a production environment has its own set of machine learning engineering challenges: How to generate these representations quickly? How to store them in a proper database? How to quickly compute similarities for production environments? In this article, I introduce two open-source solutions that aim to solve these questions: Sentence Transformers  [1] : an embedding generation technique based on textual information and; Qdrant : a vector database capable of storing embeddings and providing an easy interface to query them. These tools are applied to NPR [2], a News Portal R...
Read more

Memory Management in Flutter: Best Practices and Pitfalls

Image
  Flutter, with its promise of “write once, run anywhere,” has become the go-to framework for many developers. As with any platform, efficient memory management in Flutter is crucial to ensure fast performance, responsiveness, and a seamless user experience. While Dart, the language behind Flutter, comes with built-in garbage collection, a thorough understanding of memory management can help avoid common pitfalls and enhance app performance.  Let’s delve into the best practices and potential traps in Flutter memory management. Why is Memory Management Important?Performance: Efficient memory usage ensures faster app operation and smooth animations. Resource Conservation: Mobile devices have limited memory. Optimizing memory use ensures that other apps on the device run smoothly. User Experience: An app that efficiently manages memory reduces the chances of crashes and enhances user satisfaction. Best Practices:Leverage Dart’s Garbage Collection: Dart automatically deallocates o...
Read more

NIH autism database announcement raises concerns among researchers

Image
A new research initiative from the U.S. National Institutes of Health (NIH) seeks to gather and analyze multiple autism-related datasets to better understand the underlying causes of the condition. Letters of intent are due tomorrow, but the project’s atypical call for applications has autism scientists divided on whether or not to participate. The Autism Data Science Initiative provides $50 million for an anticipated 10 to 25 projects, according to the 27 May announcement. The funding opportunity uses an Other Transaction (OT) funding mechanism, which tends to be implemented when, among other reasons, a project is expected to evolve over time, more collaboration is required from the NIH or nontraditional research groups are involved, according to an NIH website. For a field that has recently seen budget cuts, some researchers are welcoming the opportunity for funding, says   Helen Tager-Flusberg , director of the Center for Autism Research Excellence at Bosto...
Read more