System restarting, wait...

Managing posts with Buffer, Flickr and Jekyll

28 Feb 2018 - Tag: blog

Having a blog that relies on static pages can be quiet cumbersome. This site currently runs on Jekyll, which means there is no database, no CMS, no image storage. For the most part, the site is just HTLM, CSS, JS, Markdown, Liquid and JSON files. This used to work fine when I would not post things for several months, but as I want to post more, it became clear that I needed to come up with a better sollution. What I will describe here is not a system developed to handle the limitation of Jekyll, but rather a evolving sytem that fits my needs of how I do things.

The initial plan of my blog was to host longer-form content, while social media would house regular updates and content such as photos and videos. Due to the lack of hosting space for photos and videos, I was not planning on including them into most of the posts. The process to include photos and videos into posts previously would be something like:

1. Upload images to Imgur or videos to YouTube
2. Get the embeddable HTLM
3. Open the markdown file and include the HTLM code

Even though this process is not that horrible, there are a few things that would them a real pain for me in particular.

1. I am usually working on my PixelC, so uploading images and videos is more cumbersome that uploading through a regular computer.
2. Same as 1. I am on mobile, and getting the HTLM code is usually more complicated on these platforms.
3. By preference, I try to keep markdown files clean of any HTLM. Otherwise changing CSS may require to update posts files as well.
4. Some services require more than one single Id to generate embedable code, this is specially true with Flickr. This would make it more complicated to use something like includes to handle embedded images and videos.

Due to these complications, I had made the decision to focus on uploading photos and videos to social media while using Jekyll for mostly text. When required I would include media into jekyll posts, but this would be a manual proccess.

Enter Buffer

I don’t mind posting to social media, but as time went on, I grew tired of posting to Facebook, Google+, Twitter, Instagram, LinkedIn, and the list goes on. For the most part each service does the same thing, post some text and maybe attach some images or links. Afterwards the users of each platform can interact by linking or sharing your content. Despite of this, there seems to be void of services that allow you to post to multiple services at once. On my research, Buffer was the only one that was able to post to multiple services, including Google+.

Additionally, Buffer has an API which allows us to access the list of updates(a.k.a. posts) along with all required metadata. This is a crucial part this blog because we can use it to integrate posts in Buffer within Jekyll. To do this I have generate_buffer_data.py to download all the buffer data from the last two weeks.

The next step is to merge all the posts from all the services from Buffer into a single queue. The script that does this is generate_buffer_queue.py, which will generate a hash using the post date and title. Then this hash can be used to identity the same post across all services. The result will be an array of posts that contains links to any media that was attached to the post.

Using Flickr

After the last step, we will have an array of posts which may contain links to pictures. The next step will be to check if we need to update any shared photo to Flickr. First we will use generate_flickrdb.py to use the Flickr API to generate a dictionary of photos and albums. Now I use generate_flickr_content.py to iterate over each image attached to each posts. It will identity the highest resolution picture and then it will check if the picture is already uploaded. This is done by checking if there is already a picture that contains the post hash in the tag section. Now I need to run generate_flickrdb.py to generate an updated dictionary that contains the recently uploaded photos.

Mixing Flickr, Buffer and Jekyll

At this point we have a dictionary of photos in Flickr that will have the post’s hash value in the tag. We also have a queue of posts that contains all the data of a post, including the it’s hash. The last script to run is generate_buffer_posts.py that will generate a markdown file that contains all the post information, at the end of the post there will be an include to the embedable version of the picture.

YouTube

I dont usually upload to YouTube, but I also have generate_youtubedb.py to generate a json file with all my videos and playlists. If I want to embed a video or playlist, I can simply use an include with the id of the video.

The results

After automating all these steps, I can post though Buffer to Google+, Facebook, Twitter and Instagram. Then a few times a week I can run a wrapper script to embed all the posts into the blog. This allows me to update my blog with my unfrequent social media posts. Even though I know it is not the best sollution, I really like how it works and the control I have over the entire process. I hope this was of help for you.

25 Feb 2018

I should remember that  the Central Cinema always has good movies.

18 Feb 2018

Bad UI is all around us.

15 Feb 2018

So their product leaves a residue and the fix is for the user to re apply oil to the wooden surface. Insert "you are using it wrong" comment here.
https://gizmodo.com/apple-s-homepod-can-apparently-damage-your-furniture-1823001315

14 Feb 2018

Great! I lost my yubikey again and I need it to boot my laptop

23 Jan 2018

Some pictures from the first day of the Amazon Go Store. We got some small items and thankfully the line was not too long either. Maybe this is the future, we shall see.

Amazon Post Holiday Party

20 Jan 2018 - Tags: Amazon, Party

Some pictures of our second year going to the Amazon After Holiday Party. Although this time the venue was too packed and really it was not as good as last time. :/

Another night around Capitol Hill

15 Jan 2018 - Tag: random

The weather was nice on Saturday so we decided to go out and enjoy the day.

12 Jan 2018

Friday after work. The connector stop is dead.

Getting Youtube data with Python

10 Jan 2018 - Tags: programming, python, youtube

This is done with python 2 becuase most of the Youtube DATA libraries seem to support Python 2 better than 3. There may be versions of some of the libraries for Python 3 but I did not spend anytime investigating this route.

This link goes over most of the setup process pretty well: https://developers.google.com/youtube/v3/getting-started

1. Go to the developers console and create a new project https://console.developers.google.com/

2. Now on the project page go to Credentials and create an OAuth 2 Client ID of type Other. Now download the json file which we will use to authenticate our app.

3. To install the Youtube Data library you can use pip. I got the instructions from the Youtube api-samples github page. As previously mentioned, we will use Python2.

   pip install --upgrade google-api-python-client
   

4. Now install other google libraries that will also be needed

   pip install --upgrade google-auth google-auth-oauthlib google-auth-httplib2
   

5. Now move the downloaded file and rename it to client_secrets.json, since that is what the script is expecting. I am using my_uploads.py as a reference. If you run this script now you will be prompted to visit a URL and enter the resultingauthorization code.

   cramsan@kururu ~/git/cramsan.github.io/scripts $ python2 demo.py 
   Please visit this URL to authorize this application: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=743336358314-o1grlivilob3pb4k3ra4b45kulgu20oa.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fyoutube.readonly&state=plvu3RkeSZL8Laj3wtz6IHsuvXBgY9&prompt=consent&access_type=offline
   Enter the authorization code:
   

6. This happens becuase the OAuth2 credentials previously created requiere manual copy/paste. There are four types of redirect methods and in this case we are are using the Manual copy/paste mode. Now you have everything you need to access the Youtube Data API. You can run the examples from the Youtube api-samples github page or the scrips from the repo of this blog.

Acessing the API is rather simple.

• Create the YoutubeAPI object

def get_authenticated_service():
  flow = InstalledAppFlow.from_client_secrets_file(CLIENT_SECRETS_FILE, SCOPES)
  credentials = flow.run_console()
  return build(API_SERVICE_NAME, API_VERSION, credentials = credentials)
youtube = get_authenticated_service()

• Now get the list of uploaded videos. This function will return the Id of the playlist that represents the upload list of the authenticated channel

def get_my_uploads_list():
  # Retrieve the contentDetails part of the channel resource for the
  # authenticated user's channel.
  channels_response = youtube.channels().list(
    mine=True,
    part='contentDetails'
  ).execute()
  for channel in channels_response['items']:
    # From the API response, extract the playlist ID that identifies the list
    # of videos uploaded to the authenticated user's channel.
    return channel['contentDetails']['relatedPlaylists']['uploads']
  return None

• Now that you have the Id of the uploads playlist, you can use it to iterate over every video

def list_my_uploaded_videos(uploads_playlist_id):
  # Retrieve the list of videos uploaded to the authenticated user's channel.
  playlistitems_list_request = youtube.playlistItems().list(
    playlistId=uploads_playlist_id,
    part='snippet',
    maxResults=5
  )
  print 'Videos in list %s' % uploads_playlist_id
  while playlistitems_list_request:
    playlistitems_list_response = playlistitems_list_request.execute()
    # Print information about each video.
    for playlist_item in playlistitems_list_response['items']:
      title = playlist_item['snippet']['title']
      video_id = playlist_item['snippet']['resourceId']['videoId']
      print '%s (%s)' % (title, video_id)
    playlistitems_list_request = youtube.playlistItems().list_next(
      playlistitems_list_request, playlistitems_list_response)