Web Scraping with Python

What is Web scraping?

Web scraping is a computer software technique of extracting information from websites. This technique mostly focuses on the transformation of unstructured data (HTML format) on the web into structured data (database or spreadsheet). Python has several options for HTML scraping. They are:
  1. BeautifulSoup
  2. Mechanize
  3. Scrapemark
  4. Scrapy

BeautifulSoup

Beautiful Soup is a Python library for pulling data out of HTML and XML files. It works with your favourite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree . It helps you pull particular content from a webpage, remove the HTML markup, and save the information. Professionals can scrape information from web pages in the form of tables, lists, or paragraphs. Urllib2 is another library that can be used in combination with the BeautifulSoup library for fetching the web pages. Filters can be added to extract specific information from web pages . Urllib2 is a Python module that can fetch URLs. It commonly saves programmers hours or days of work.

Mechanize

Mechanize A very useful python module for navigating through web forms is Mechanize. It acts like a browser allowing you to do web scraping , functional testing of web sites and things no one has thought of yet.

Scrapemark

Scrapemark is a super-convenient way to scrape webpages in Python. It utilizes an HTML-like markup language to extract the data you need. You get your results as plain old Python lists and dictionaries. Scrapemark internally utilizes regular expressions and is super-fast.

Scrapy

Scrapy is a free and open source web crawling framework for large scale web scraping , written in Python. It gives you all the tools you need to efficiently extract data from websites , process them as you want, and store them in your preferred structure and format.




Related Topics
  1. Python Interview Questions (Part 2)
  2. Python Interview Questions (Part 3)
  3. What is python used for?
  4. Is Python interpreted, or compiled, or both?
  5. Explain how python is interpreted
  6. How do I install pip on Windows?
  7. How do you protect Python source code?
  8. What are the disadvantages of the Python?
  9. How to Python Script executable on Unix
  10. What is the difference between .py and .pyc files?
  11. What is __init__.py used for in Python?
  12. What does __name__=='__main__' in Python mean?
  13. What is docstring in Python?
  14. What is the difference between runtime and compile time?
  15. How to use *args and **kwargs in Python
  16. Purpose of "/" and "//" operator in python?
  17. What is the purpose pass statement in python?
  18. Why isn't there a switch or case statement in Python?
  19. How does the ternary operator work in Python?
  20. What is the purpose of "self" in Python
  21. How do you debug a program in Python?
  22. What are literals in python?
  23. What is Python's parameter passing mechanism?
  24. What is the process of compilation and Loading in python?
  25. Global and Local Variables in Python
  26. Is there a tool to help find bugs or perform static analysis?
  27. What does the 'yield' keyword do in Python?
  28. Comparison Operators != is not equal to in Python
  29. What is the difference between 'is' and '==' in python
  30. What is the difference between = and == in Python?
  31. How are the functions help() and dir() different?
  32. What is the python keyword "with" used for?
  33. Is all the memory freed when Python exits?
  34. Difference between Mutable and Immutable in Python
  35. Explain split() methods of "re" module in Python
  36. Accessor and Mutator methods in Python
  37. How to Implement an 'enum' in Python
  38. Important characteristics of Python Objects
  39. How to determine the type of instance and inheritance in Python
  40. How would you implement inheritance in Python?
  41. How is Inheritance and Overriding methods are related?
  42. How can you create a copy of an object in Python?
  43. How to avoid having class data shared among instances in Python?
  44. Static class variables in Python
  45. Difference between @staticmethod and @classmethod in Python
  46. How to Get a List of Class Attributes in Python
  47. Does Python supports interfaces like in Java or C#?
  48. What is used to create Unicode string in Python?
  49. Difference between lists and tuples in Python?
  50. What are differences between List and Dictionary in Python
  51. Different file processing modes supported by Python
  52. How do you append to a file in Python?
  53. What are the differences between the threading and multiprocessing?
  54. Is there any way to kill a Thread in Python?
  55. What is the use of lambda in Python?
  56. What is map, filter and reduce in python?
  57. Is monkey patching considered good programming practice?
  58. What is "typeerror: 'module' object is not callable"
  59. Python: TypeError: unhashable type: 'list'
  60. How to convert bytes to string in Python?
  61. What are metaclasses in Python?