freestuffs package

Submodules

freestuffs.city_list module

Craiglist friendly city names.

Run this script inorder to print out a dict of Craiglist city names, for use in creating a valid url.

Attributes:
  • CITIES – A dict of cities, with the human-friendly

    name as the key and url friendly name as the value.

freestuffs.city_list.scrape_cities()

Scrape city and link for dict of valid names.

freestuffs.stuff module

This houses the Stuff class.

Use stuff_scraper in order to gather a list of stuffs. For testing, the reverse Geolocator is at nominatim.openstreetmap.org.

Example usage:

>>> from stuff_scraper import StuffScraper
>>> stuffs = StuffScraper('montreal', 5).stuffs
>>> print(stuffs[0].thing)  # Title
Meubles / furniture
>>> stuffs[0].find_coordinates() # pass precise=True in constructor
>>> print(stuffs[0].coordinates) # to automatically fetch coordinates
['45.617854', '-73.633931']
class freestuffs.stuff.Stuff(thing, url, location, image, user_location)

Bases: object

A freestuff Craigslist object.

Fill this object with the information from a craigslist page. (There is no price attribute, because it is designed for invaluable things). The precise coordinates are not initially set, because they require significantly more requests. See class method get_coordinates().

Attributes:
  • thing – title of object passed explicitly
  • url – constructed from url, implicit
  • image – passed explicitly
  • user_location – passed explicitly, requires clean up
  • coordinates – array of longitude and latitude
Keyword arguements:
  • thing –
  • url –
  • location –
  • image –
  • user_location – must conform to valid craiglist url
find_coordinates()

Get and set longitude and Latitude

Scrape individual posting page, if no coordinates are found, cascade precision (try location, try user_location, or set to zero). Returns an array, first latitude and then longitude.

freestuffs.stuff_charter module

Chart where free things are.

The StuffCharter class is a wrapper around the folium openstreetmap python object, which in turn generates a leaflet map.

Example usage:

>>> from stuff_scraper import StuffScraper
>>> from stuff_charter import StuffCharter
>>> stuffs = StuffScraper('montreal', 5, precise=True).stuffs
>>> treasure_map = StuffCharter(stuffs)
call save_map(path) generate html map
>>> treasure_map.save_test_map() # saves map in current dir
BEWARNED, this map is likely inaccurate:
Craigslist denizens care not for computer-precision
class freestuffs.stuff_charter.StuffCharter(stuffs, address=None, zoom=13, do_create_map=True, is_testing=False, is_flask=False)

Bases: object

Post folium map of freestuffs.

After constructing Mappify map object, call create_map and pass in map_path in order to create the HTML map.

Attributes:
  • treasure_map – an OSM folium map object
  • stuffs – list of free stuff
  • user_location – the user’s location
  • start_coordinates – the origin coordinates for city
  • zoom – default map zoom
Keyword arguments:

  • stuffs – a list of stuff objects

  • address – for an optional map marker of the user address.

  • do_create_map – set to False to override modify attributes

    before create_map.

  • is_testing – use to test module from commandline

  • is_flask – automatically create map for treasure-map

  • zoom – the map default zoom level

add_address(_address)

Add address to folium map

create_map(is_testing=False, is_flask=False)

Create a folium Map object, treasure_map.

treasure_map can be used to save an html leaflet map. This method is called automatically on __init__ unless do_create_map is set to False.

Keyword arguments:
  • is_testing – creates a map in webmap directory
  • is_flask – creates a flask map
find_city_center(location)

Return city center longitude latitude.

save_flask_map()

Create html map in flask server.

save_map(map_path, css_path=None)

Create html map in map_path.

Keyword arguments:

  • map_path – the path to create_map in

  • css_path – the path to override css

    (defaults to bootstrap via folium)

save_test_map()

Create html map in current directory.

Should have python -m http.server running in directory

sort_stuff(stuff)

Return a color according to regex search.

  1. Furniture pattern, red
  2. Electronics pattern, blue
  3. Miscellaneous pattern, black
  4. no match, white

sort_stuff will return with the first pattern found in that order.

TODO:
  • Set and patterns as modifiable attributes.

freestuffs.stuff_scraper module

This module is a Craigslist scraper.

Example usage:

>>> from stuff_scraper import StuffScraper
>>> stuffs = StuffScraper('montreal', 5).stuffs
>>> print(stuffs[0].thing) # Print title
Meubles / furniture
class freestuffs.stuff_scraper.StuffScraper(place, _quantity, precise=False, use_cl=False)

Bases: object

The freestuffs Craigslist scraper.

Compile parrellel lists of stuff attributes in order to store a freestuffs list, with an option for including stuff coordinates.

Attributes:
  • stuffs – a list of stuff objects
  • soup – bs4 soup of Craiglist page
  • place – the city to search, in Craigslist friendly format
  • locs, things, images, urls – stuff attributes lists
  • quantity – how many stuffs gathered
Keyword arguments:

  • _quantity – how many stuffs to gather

  • precise – A boolean to explicitly use geolocator

    and crawl individual posting URL

  • use_cl – user input for place

get_images(_soup)

Scrape images.

Uses wikpedia No-image image if no image is found.

Keyword arguments:
  • soup - bs4 object of a Craiglist freestuffs page
get_locations(user_place, _soup)

Scape locations.

Returns a list of locations, more or less precise. Concatnate user_place to string in order to aid geolocator in case of duplicate location names in world. Yikes.

Keyword arguments:
  • user_place – the city, in Craigslist format
  • soup – bs4 object of a Craiglist freestuffs page
get_things(_soup)

Scrape titles.

Keyword arguments: - soup - bs4 object of a Craiglist freestuffs page

get_urls(_soup)

Scrape stuff urls.

Keyword arguments:
  • soup - bs4 object of a Craiglist freestuffs page
refine_city_name(user_place)

Refine location for two-word cities.

setup_place()

Take cl input of user location.

freestuffs.treasure_script module

Module contents