This is the first time I face a scraping project, but I recently had some ideas that require scrapping content and at some point I should be working on this so...

This whole thing came up from my need to find new piano pieces that I could practice according to my level. If you have ever played an instrument for some time, you’ll probably felt the same way as me.

You may know tons of pieces you’d like to play, aside the “boring” repertoire we all know to improve, but don’t know enough music yet to understand at a first glance if they are way above your level, or is something affordable you can propose your teacher.

DISCLAIMER: Let me clarify that above I wrote “boring” between quotes because, even some of these pieces may seem not very interesting to learn, you should always listen to what your teacher recommends.

I know there are different scales that grade musical pieces into different levels (ABRSM, Henle Verlag, Royal Conservatory of Music, ...) taking into account different aspects, I particularly like the explanation from 

 because it has a marketplace with enough piano works to give it a try to this scraping thing. Also, as this is just a pet project for my own I wasn’t interested either in scraping the whole 

 page to present the same content with a different format.

I think what they do with that page is a super valuable work, and strongly recommend any piano enthusiast to visit their website, because you’ll find also 

 (another amazing project) links on it to download for free the pieces.

, a Python package widely adopted for scraping, but as I’m not used to Python package management (pip/virtualenvs always felt a bit magic for me), I decided to go with JS to develop faster, as I’m very used to it in Factorial.

Once I chose the language, I started checking some scraping packages and scrape-it showed up outta nowhere. Being maintained for several years (released on 2016 and last version from Aug 2021), having a good amount of downloads, and a lightweight unpacked size made it a perfect candidate for what I was looking for.

And tbh it took me less time than I expected to start scrapping the exact information I wanted. The main function I use to scrape content receives a url (obviously) and a schema with the content to be retrieved from the page. This schema can include individual elements, or nested elements if you want to scrape the same information for a set of children elements (exactly what I needed). Just indicating the selector rules to match the parent node (listItem is the naming convention for scrapeIt) and the rules to match the information you want from their children, you are good to go.

import scrapeIt from 'scrape-it'

const workCollectionSchema = {
	workCollection: {
    listItem: '.result-item.clearfix',
    data: {
      title: {
        selector: 'h2.main-title',
      },
      author: {
        selector: 'h2.sub-title',
      },
      href: {
        selector: 'a.cover-wrapper',
        attr: 'href'
      }
    },
  }
}

const url = 'https://www.henle.de/en/search/?Scoring=Keyboard+instruments&Instrument=Piano+solo'
const res = await scrapeIt(url, workCollectionSchema)

console.log(res.data.workCollection) // [{ title: ..., author: ..., href: ...}, {...}, ...]

It’s important to ensure an ethical scraping, for that reason I added a delay of 10 seconds between each scrape to ensure the stability of Henle page is not being compromised by my information retrieval. Apparently there is no standard about this, but as Henle didn’t have any specific robots.txt information on scraping, and people in different threads suggest using something between 5-15 seconds, I thought it would be a good idea to do so.

Also, I think is extremely important to credit the different creators from which I’ve learned something, or made my development faster while working on this project. For that reason, I want to thank the following libraries / people / organisations:

: for a fantastic explanation of the difficulty scales available for piano pieces.

: for owning all the content I extracted to make this pet project possible.

: creator of the scrape-it package (among others) for making my first scraping experience extremely smooth.

There are many libraries to scrape content, but scrape-it makes it so simple. I had no prior experience to scrapping and it took me just some minutes to extract the information I needed.

I discovered scrape-it and it is incredibly easy to use

On September 2021 I started one of the biggest projects I have worked on so far, not just in Factorial, but in my whole developer career. We had been redefining our new brand for months and the implementation phase was finally starting!!

We decided to tune our design system in desktop application, mobile app, public pages and our blog to be “brand friendly”. Guess who was the dev that was working on his own (aimed by designers and the brand team of course) to complete that “simple” task... 😅

Whatever, I’m deviating from the original topic. 

 One request that brand team had for the rebranding was that we change the login process on our desktop application, which currently is a bouncing logo animated with CSS. Which is fine so far, but a big change like what we were about to build needs a proper presentation for the user.

I talked about this with a Motion Designer (a completely new role for me) from Brand Studio team and he told me that he can generate some animation for me and bring me the file. And at first I thought he was talking about a GIF file, and I started wondering about things like how it would look like on different screen sizes, how to compress it to avoid delivering a monstrous file, ... But he send me a JSON file and I was like wtf 🤨 

Here I discovered that Lottie files are really a thing, and they remove several layers of friction between designer and developer when working with animations:

The designer delivers a complete animation so 

the developer won’t spend a second touching any kind of CSS

. This means that the original design and the final result are exactly the same.

 over the animation. From code you can decide when to play, stop, rewind, adjust the speed, play on loop, ... This opens a world of possibilities to improve UX in your website, by providing interactive and more engaging blocks of information.

 (something I haven’t tried yet) in gzip format. This makes it easier to deliver more complex lottie animations without adding much overhead to some page. → Despite this, it is always recommended to work with short animations, because for longer ones the size increases a lot.

The code to load an animation is quite simple actually, I mostly copied 

this answer from user Danila in StackOverflow

. You only need three main things to make this work:

 package → You can install using npm or yarn

Have a lottie animation → You can find many free to use in 

Create the element in which your lottie animation will be loaded → In my case I’m using a simple div with some width restrictions to avoid flickering.

import type { LottiePlayer } from 'lottie-web'

export const LottieLoader = ({
  animationName,
  loop = true,
  autoplay = true
} : Props) => {
  const ref = useRef(null);
  const [lottie, setLottie] = useState<LottiePlayer | null>(null);

  useEffect(() => {
    import('lottie-web').then(Lottie => setLottie(Lottie.default))
  }, []);

  useEffect(() => {
    if (lottie && ref.current) {
      const animation = lottie.loadAnimation({
        container: ref.current,
        renderer: 'svg',
        loop: loop,
        autoplay: autoplay,
        path: `/lotties/${animationName}.json`
      });

      return () => animation.destroy();
    }
  }, [lottie, animationName, autoplay, loop]);

  return (
    <div ref={ref} className={cn(
      'items-center',
      configWidth[animationName] // To avoid flickering
    )}/>
  )
}

This post is created thanks to many people, so I want to thank them individually:

 for showing me the world of Lottie animations. Check his profile, his works are simply amazing.

 for posting a perfectly explained code that saved me some time dealing with possible bugs.

 for creating and uploading the two amazing Lottie files I’m using in my personal website.

You probably found in the homepage a lossless quality waving animation. This is not a GIF, a video or a CSS animated image. It is a lottie animation that you can start using without too much setup.

Lottie-web to deliver awesome animations in your website

I decided to build myself a personal website as a first step to start working regularly on side projects. Also, I don’t feel like updating my CV every now and then because summarising all your experience and highlight things in a more custom way is hard when you are bounded to do it on a simple PDF file...

I wondered for days how could I build a different website, something that doesn’t look so empty like “I’m Sergio and this is my experience, that’s all folks” (if you website looks like that don’t take this post personally 😅 ). At some point I found out the 

 personal page which had a TIL (Today I Learnt) section with small pills of information that he was learning from time to time and that can be useful for others. That’s a great idea!! So I started to build something similar.

I realised from the very beginning that this web should be server-less, as there is no logic aside from handling content that I could be storing on a CMS. In Factorial we use 

 for that, a good option to build kinda complex structures, and manage content for a bunch of different users. Unfortunately, I’m not a big fan of their API docs so I decided to give 

 a try and see if it could be a suitable alternative.

Different tools, but the same pains IMO... I found myself spending about half an hour to define the models I wanted for this website, e.g. a Page, a TextSection, a MarkdownSection, ... Again I had to face some not very clear API docs, but not really a big deal. Once everything was setted up, I faced the most important problem, which made me red-flag it and look for another solution: the friction to generate content fast and all at once. I had to create one component every time, write in editors that didn’t provide the best experience, if I wanted to change some block I had to remove it, but copying things before-hand...

As a matter of casualty that week I read a very interesting message related to the brand new Factorial engineering blog. Some dev was explaining the approach they followed and it was exactly what I needed:

“Any amount of friction that gets in the middle of writing a blog article will be used an excuse, so we’ve thought that using Notion as a repository looks like a great idea.”

Notion API was in beta, but it looked powerful enough to build a simple website like this so I decided to give it a try. Best decision ever.

Probably this post requires some more code fragments than others, because connecting to an API, specially to a CMS and retrieving/handling the content from it is not trivial. Is not hard, but it can’t be done it in seconds (at least I can’t), if you follow the guides, you should find almost no problems.

First of all, you’ll have to get familiar with the concept of 

 that Notion uses internally, because these are the places where you can store your content. In fact they are nothing more than easy structures like tables, boards, or timelines (there are more, 

). Some pseudocode to fetch information from these database would look like this:

import { Client } from "@notionhq/client"

const fetchNotion = async (
  query: Query,
  handler: any
) : Promise<ValidFetchResult> => {
	const notion = new Client({ auth: process.env.NOTION_KEY })
  const response = await notion.databases.query(query)
  
  return response.results.map(handler)
}

The steps are more or less the same that you need to follow in order to fetch any other API:

You start the connection with Notion client using your 

You use a handler to sanitise the results.

The queries are not hard to understand either, below you can find an example of some query:

const getPageBySlugQuery = (slug: string) => {
  return {
    database_id: process.env.DATABASE_ID,
    sorts: [
      {
        property: 'Publish date',
        direction: 'ascending',
      },
    ],
    filter: {
      property: 'Slug',
      text: {
        equals: slug
      }
    }
  }
}

An easy to understand query that anyone could understand without even reading the function name. All queries are oriented towards a specific database, so you have to provide its ID.

The only “problem” I found while using Notion API is that querying an item from a database doesn’t return the “Page content” with it. If you’ve ever used Notion (if you hadn’t give it a try), when you create an item inside a database it has some properties that you define (those above fold when you open the page), and then the rest of the content (below fold) that includes any kind of Notion block to format your content as you want.

If you want to access the content of some database item you’ll have to do two queries (at least this is what I had to do), one to obtain the item that you are looking for, and another one to retrieve the content using that item ID → 

If you know another way to simplify this process, pleas ping me, I’d love to know more about this 

const handler = (pageItem: any) => {
  const notion = new Client({ auth: process.env.NOTION_KEY })
  const pageContent = await notion.blocks.children.list({
	  block_id: pageItem.id
  })

  return {
    content: pageContent.results
  }
}

Notice that the query in this case is a bit different, we search for 

, and also the endpoint we use from the Notion client in this case is 

. Small differences, but doesn’t add a lot of complexity, though it would be great to do this operation all at once of course.

 for releasing an API which I can use for free (at least for now I think 🤔 ) and is a great option for short-term content management.

, that posted the idea to use Notion API as the base for our blog. Not going to lie, how Factorial engineering blog was built is a great inspiration for this website. Despite that, I built most things here before the blog was launched, so is very far from being a copy/paste 🙂

 for inspiring me to copy this idea of TIL, hope he don’t mind 😂

Even though Notion API is in beta status it already offers stable and easy tools to connect with your workspace and retrieve page content. Actually, this website is built upon Notion. 

CMS hosted website using Notion

There is not much pretext in this post. I was just building this website and wanted to add some section with links in the homepage, so anyone that wants to ping me know how. I didn’t want to add all my social networks, but just the main ones like LinkedIn, Github and my contact email, a quite easy task you may think. In fact it is, but my OCD with image formats just showed up and started to wonder about things like the image size, how it scales among different screen sizes, ...

So I decided to avoid downloading and using a PNG or JPG image and look for some SVGs which are more lightweight and scale better. But then I realised that is not always trivial to  Of course it didn’t took me very long to find 

It’s amazing how you can have all the icons you need by installing a single npm package.

In just a few lines of code I am importing the SVG icons that I need from 

, and adding custom styles to them without any struggle.

import { IconType } from 'react-icons'
import { FaGithubSquare } from 'react-icons/fa'
import { FaLinkedin } from 'react-icons/fa'
import { GrMail } from 'react-icons/gr'
import { IconContext } from 'react-icons'

const socialLinks = [
  {
    href: 'https://github.com/SergioMD15',
    icon: FaGithubSquare
  },
  {
    href: 'https://linkedin.com/in/sergio-mosquera-dopico',
    icon: FaLinkedin
  },
  {
    href: 'mailto:sergiomosquera@gmail.com',
    icon: GrMail
  }
]

const SocialIcon = ({ href, icon } : SocialIconProps) => {
  return (
    <Link href={href} passHref>
      <a>
        <IconContext.Provider value={{ size: '36px', color: '#111827' }}>
          {React.createElement(icon, null)}
        </IconContext.Provider>
      </a>
    </Link>
  )
}

 repository for making such an easy to use and helpful library.

Adding new images into project is not always easy, specially if there is no designer that delivers the assets you need. In those cases, you rely just on the power of Internet and its free resources. Learn how react-icons library has tons of free resources that you can use instead of spending hour searching for free SVGs

Hey!