 Developed Puppeteer eBook crawler with React Nodejs Express Heroku



[deployed here.](https://ebook-crawler.herokuapp.com/)

I've built a Puppeteer webcrawler.
wanted to a one click browser which only crawling all eBook platforms on South Korea.

There are 3 major eBook services. Millie of Archive, Ridiselet from Ridibooks, Bookclub from Yes24.



It took 4 days. Everyday around 10 hours coding... Still need improve my debugging. Maybe I'm going to try `vim` for my next IDE. Saw an youtube video which compares that productivity with `VScode`. 

## Obstacle 1 : Didn't know how to set the server side.

I didn't know how to set up the Server side with pure nodeJs, as I only made python or java website only. So, Had struggled to find out 'well going package'.

<iframe width="560" height="315" src="https://www.youtube.com/embed/8bNlffXEcC0" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

But I'd recommand follow up above package, as I got smoothly success on well running server side. Although I couldn't ensure how to remove the `view` part which set the `view engine, jade`. So that misled me a little time wasting. So I'd recommand to follow below guy's comment. However, his explantion is very neat and simple. ( I'd like to have that level of coding, soon! )

![remove-view-part-because-using-react](https://user-images.githubusercontent.com/35059428/71762597-77900500-2f14-11ea-944f-e9f4434f9d65.png)

but  struggled with the front end `proxy` setting part.
Frontend part should have to connect with backend when it request. As many as documents say 'just write it in `package.json`, `"proxy" : "localhost:4000"` however, I wasn't that lucky one.

So I found another solution.

<script src="https://gist.github.com/sj-log/6de6e5d3cb70bfab7feade2f588ecba6.js"></script>

then simply install `http-proxy-middleware`
with package manager.

this code should be inside of `src` in `client` side.

## Obstacle 1.5 : How to react fetching the crawled data?

[you can check the code in here.](https://github.com/sj-log/eBook-crawler/blob/master/react-backend/client/src/App.js)

I used `axios`.

`input tag` got a typing, syncronizely changing the state value through `onChange()`. 
when a `form submit` fires, it delivers the sign to `onSubmit()` function, which fires a fetch by `axios.get(/search?inputData=${inputData})`.
then It delievers to the crawling function in server.


## Obstacle 2 : How to make puppeteer pages crawling  more than just one 


Puppeteer should be set up.
Well, firstly I compared with `PhantomJs`, and `selenium`.
but since I've got an advise which `Puppeteer` is kinds of superior one, decided this one.

After the complete with 1 eBook platform crawling, had to find out how to run Puppeteer in pararell.
but found [`Puppeteer-cluster` was the solution.
](https://github.com/thomasdondorf/puppeteer-cluster)

<script src="https://gist.github.com/sj-log/ff437e22da2577dc47cb07c57fc45e33.js"></script>

the crawling fired by `await cluster.execute()`, with the `resolve` return value as well.
It's better to put this in mind.

<iframe width="560" height="315" src="https://www.youtube.com/embed/O5yBkHjnOvo" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>


## Obstacle 3 : The thumbnail is crawlable when client look that part.

Yeah, the img what I want to pulling wasn't happen very well,[ because the imgs were only clickable when it's passing through by scrolling.](https://select.ridibooks.com/search?q=%EA%BF%88&type=Books)

### Solution

so I should had to find out the automatically scroll down until the body bottom.

<script src="https://gist.github.com/sj-log/249f92c81c79acfee4ee1288499f1ea4.js"></script>


## Last Obstacle : Deploy on Heroku

[Found from here.
](https://github.com/puppeteer/puppeteer/blob/master/docs/troubleshooting.md#running-puppeteer-on-heroku)

the cluster should be initialized as below. `  args: ['--no-sandbox']`
<script src="https://gist.github.com/sj-log/e18a78265227eca3d67cb8d79422dc80.js"></script>

This was unexpectedly difficult part, because I followed [their suggested package](https://github.com/CoffeeAndCode/puppeteer-heroku-buildpack), but didn't work. Rather just [original one](https://github.com/jontewks/puppeteer-heroku-buildpack) was totally fine to go.

The above misleading made me waste a few hours by `chrom-linux  errors`.

[you can get more info here](https://timleland.com/headless-chrome-on-heroku/).

got a problem with error `code 14`.


but by adding `web: npm run start` on the Procfile, `heroku buildpacks:set heroku/nodejs`,`heroku buildpacks:add https://timleland.com/headless-chrome-on-heroku/`,`heroku ps:scale web=1`

made all done clearly. 😊


# After all

during this job, I found the UI framework could be a shortcut to build whole frontend jobs.
The Ant artist framework makes very neat and white ton styling that I like to make.

But still need handle the `Book not found result`, and reponsive `img`.

Hope these footages make your pain  shorten!

  [deployed here]https:ebookcrawlerherokuappcom  Ive built a Puppeteer webcrawler wanted to a one click browser which only crawling all eBook platform

Developed Puppeteer eBook crawler with React Nodejs Express Heroku

Jan 04 2019