Go back

How I made Enigma Drive

Engineering Favourites
6 months, 3 weeks ago

I got this idea of Enigma Drive (Which is just like Google Drive, OneDrive, or Dropbox) when I was having a day trip to Whistler. As I saw the views of the Sea to Sky Highway, I suddenly got an idea. What if I make a server on my Raspberry Pi (A cheap computer), that receives the bytes of a file, and write it into a file, and an Apache or
Nginx (They are both a server software.) server so it will host the files? I pondered the idea a bit more and also concluded that there could be an existing software that I could use for this project.

When I came back, I began to search for the existing repositories on Github, and I came across this project named "SeaweedFS".

My requirements for the file servers that I was going to use was:

- Must support TLS(SSL)
- If I was going to access the server on the public, open, internet, it would be nice to keep things private then having all our files exposed!
- Must serve files
- It would be nice to see files online.
- Preferably, have a nice and easy to use Python SDK.
- Would be easy to implement file I/O when there is an SDK.

SeeweedFS seemed perfect, however, I later found out that it didn't support TLS. (Or so I thought). I began again on my great search of repositories and I came across...

Minio!

Minio met all of my requirements. It was a perfect choice. I quickly wrote out some code to test my Minio on my Raspberry Pi:

from minio import Minio
from minio.error import ResponseError
import urllib3

http = urllib3.PoolManager(
cert_reqs='CERT_NONE')

# Initialize minioClient with an endpoint and access/secret keys.
minioClient = Minio('192.168.1.70:9000',
access_key='My access key',
secret_key='My super secret classified key',
secure=True, http_client=http)

try:
# print(minioClient.fput_object('drive', 'test.png', './photo.png'))
print(minioClient.presigned_get_object('drive', '1'))
except ResponseError as err:
print(err)

print(minioClient.list_objects_v2('drive'))


It worked! I self-signed my certificate for Minio for HTTPS connection and got started on coding my drive.

...

Ah ha! My drive was complete. I uploaded my code to Github (https://github.com/hillcrestpaul0719/django-girls), and I went to PythonAnywhere(My website hosting provider) to download my code and to make Enigma Drive Live. Everything went smoothly until I tried to upload my file. I checked the error logs, and it showed that multithreading was not allowed in PythonAnywhere Web app environments. My code used multithreading to upload the files the user uploads to my server. PythonAnywhere apparently didn't allow multithreading, claiming that it could easily cause resource hungry process. So I researched and found that multiprocessing was allowed after all. I switched my upload code to multiprocessing, and I updated my website. I uploaded my file. My file didn't show up! I looked at the logs. The error was that my web server couldn't reach my Minio Server. What gives? I could access my server outside of my own network!

After a long research, I found out that PythonAnywhere blocked access to website except for whitelisted sites. Of course, my Minio Server wasn't on the list. If I paid PythonAnywhere, they would let me access those sites. THAT WAS IT! I was moving to Heroku(Another hosting provider)!

Heroku had a unique configuration setup. Moreover, it didn't have a persistent disk storage! That meant that my database would be wiped every day or so. I used SQLite, which meant that I couldn't host my database somewhere else (I could have used SQLitening, but it was just making things more complicated). If I needed a persistent disk storage, Heroku said that I would have to purchase a hosted block storage like Amazon S3. I concluded that I would have to pay some money after all, and because I don't have any experience with Amazon and Heroku, my only choice was my dear old friend PythonAnywhere.

I looked at the pricing for PythonAnywhere. I was using a free account. The cheapest option was $5 per month. $5 was okay with my allowance. I purchased it.

The paid option also came with the option of using my own domain to access my website. I created CNAME record for my domain, which was "www.enigmatic.network" and I pointed it to my website on PythonAnywhere. When I visited through my domain, I immediately saw a certificate error! I realized that the certificate I was using was only for PythonAnywhere sites. I dug through the PythonAnywhere forum and found a tutorial to Use ZeroSSL(A service that provides free certificates). I followed the tutorial and submitted my certificate to PythonAnywhere support, and waited.

Meanwhile, I wanted to check if my drive worked. I visited Enigma Drive. Everything worked fine, except that couldn't download the files I uploaded to Enigma Drive. I was getting a certificate error when I tried to download my files. Certificate error?!?! I remembered that I self-signed my certificate for Minio. I followed the tutorial again and generated a wildcard certificate. I put the certificate files onto my Minio Server, and I ran the server. I got a certificate error again. Isn't wildcard certificate supposed to match all of my subdomains? I dismissed the problem for the next day, and I called for the night.

The next day(Which is the day of this writing!) PythonAnywhere sent me an email that they applied the certificates I sent them.

For my certificate error the day before, I found that a wildcard certificate only matched one-level sub-domains. My Minio server was at "w.ca.rasgma.enigmatic.network". I again got my certificates for my Minio server, and it worked!



The Future


When I was developing Enigma Drive, I realized that the storage could run out. Currently, I decided to provide unlimited storage for my users. If my current 16GB USB stick runs out, I can just transfer my user's files onto a bigger storage medium and use that as my new storage. Now Enigma Drive is future-proof!

Check Enigma Drive out here: https://www.enigmatic.network/drive/

Comments?
Questions?
Complements?
Roasts?

Why not comment below?!?!



Comment

Name:



Content: