Call for paper and notable conferences

 Unisys India's Cloud 20/20 Version 2.0 is an online technical paper contest. 

The topics being focussed on are 

1)     Automation for Cloud Computing

2)     Virtualization (server, storage, networks)

3)     Application Development for the Cloud

4)     Moving Workloads from Datacenter to the Cloud

5)     Consumerization of IT

6)     Cloud Computing for Airports – Solutions and Benefits

The contest is for

1) Post-graduate with focus on computer science and related branches

2)  Pre-final & final year engineering students (BE/B.Tech in CS, IT and related branches) 

   The site says that the students submitting papers could have career opportunities at Unisys.

Last date for submission of abstract is 12th December.

The Computer Society of India’s International Conference series on Software Engineering (CONSEG)-11 has the theme “Software Quality- the Road Ahead”

The conference will have both invited and contributed papers. World leaders in this area are delivering key note talks on advanced topics. These will be combined with Plenary Talk by Microsoft and Invited talks by stalwarts from Indian IT companies. Presentation of Talks and by Posters of contributed papers is also planned.

Some notable speakers are::

Alain Abran,   Professor ,   University of Quebec	A Maturity Model of Software Product Quality
Bill Curtis, SVP and Chief   Scientist, CAST	Measuring and Managing the Non-Functional, Internal Quality of IT Software: The Next Wave Hitting Application   Customers and Suppliers
Dan Galorath, CEO,   Galorath   Consulting	More Successful Projects With Viable Estimates: A 10 Step Estimation Process With Emphasis on Parametric Estimation
K S Trivedi Professor,   Duke University	Software Aging and Rejuvenation
Murali Chemuturi,   Author and Consultant	Requirements Management

Microsoft interview and written round

written rnd for MS had 5 quest ::

1) a simple quest to debug a given node from linklist :: remember dont rewrite the whole code again .. point out bugs and right code for tat

2) 2nd quest was a given code to find output, it ws the recursive LCS problem

3) 3 rd was the code to find an element in rotated array

4) 4th was to write test cases for server client 

   like client sends req : GET file1.txt  and server responds by sending file 

 5) Design a mobile application to allow blogging experience on the phone 

 Its always better to ask quest frm them to clarify as I did in 5th quest

Interviews::

1st interview 

 1)project quest .. project was antispamming so he asked abt ways to SEO

 2)2 arrays given first of size >(M+N-1) with M elements filled up 2nd array has N elements , both sorted .. return merged sorted array  "test it"

 3)Program Manager question : ways to improve Ranchi , my client is a Municipal Corporation Commisioner

2nd inerview

 1)spiral array print 

 2)write code to see if two rectangles overlap : simple rectangles, sides parallel to x or y axis 

 3)iterative tree traversal

3rd interview

 1)Project again : Anti-Spamming

 2)Given an array of size M*N , the place where 0 is present , say A[i,j] is 0 then make all elements of row i and all elements of col j 0.. for all 0s present in original array  "test it"

4th interview

 1)implement stricmp (ignore case strcmp)

2) 100 Kb block of memory given ... you have to allocate queues from the memory such that 

  - number of queues are dynamic

  - size of queues is dynamic

  - use 90% of space effeciently

  make class  for queue with behavior

                       getByte(char c) :: get a byte from memory and put char c there

                       putByte() :: send a byte back to memory                              

  I was finally chosen for Developer's profile 

Google Scribe : An autocompletion service

Google Scribe was launched this september. Its beta phase looks deficient but promising. I had the notion of such a auto completion of text . Imagine yourself typing at a document and the autocompletion service saving your time. I am trying to extend the same idea over to Rhyme - Seeker ,"a ranked autocompletion of poetry". I saw a similar idea at ISI for Creative language use. I believe that the technology of Scribe could be extended to achieve the task.
       Google Scribe seems to be straightforward application of web n-gram language models covered in an AJAX interface. Some of its mistakes demonstrate the drawbacks of not utilizing long range word dependencies and topical context. It also adds numerous options for searching.

  Google states, “Scribe’s suggestions indicate correct or popular phrases to use". Most probably , as I aslo experimented with it, Google Scribe doesn't personalize the autocompletion yet. It would be amazing to see scibe getting enhanced and added to google docs .

The new Lucene Based Search Architecture of Twitter

A new backend search architecture that was launched few weeks ago. The previous system was based on the original Summize search system that Twitter acquired in 2008. The old system was based on MySQL. This system became difficult to scale for such large data. They turned toward a new, modern search architecture that is based on a highly efficient inverted index instead of a relational database, which is the obvious choice for large data indexing for quick search.
                   They did lots of tuning for improving the current version of  Lucene. Some of the highlights of the  changes include:

• significantly improved garbage collection performance
• lock-free data structures and algorithms
• posting lists, that are traversable in reverse order
• efficient early query termination 

We can hope to find these contributions to the Lucene soon.

According to the blog, the new search system was designed to handle over 1,000 TPS (Tweets/sec) and 12,000 QPS (queries/sec) = over 1 billion queries per day . Besides the challenging query volume, the data needs to be available quickly, a tweet needed to be searchable in less than 10 seconds.

The main benefit to users is that the new system is much more scalable and can support an index that is twice as large as previous versions which means that you can search for tweets further back in time.

Tejas Networks Software Written and Interview

The written part of Tejas 4 sections
1:Apti 10 questions
2-C apti
3- General Comp. Sc. : Networking, OS, compiler
4-Coding Section

Time:2 hrs
All sections had their sectional Cut-off

4-The Coding section consisted of 4 questions:
*Find the maximum Height/ Depth of a binary tree
*There are n stations numbered from 1 to n stored in an array
the distance between any two stations is given as::
D[i][j]=0, if no path exists between the stations i and j
D[i][j]=n , if the distance between two stations i and j is given to be n
The stations are arranged in a ordered linear sequence , like S1,S2,S3......Sn
*Given a tree store it in a file and retrieve it back from the file
*A file is given with many 0s stored in continuous way  , store it in another file such that when you store try saving the space by using minimum amount of space. When you want to create the original file , you should be able to do so with the new file created. Hint was given so as to utilize the lseek function (a fuction in the Unix)

3-General Comp. Sc. : Networking, OS, compiler
*Write down the Java Socket APIs
*What is deadlock? How do you prevent it?
*What is IPC . GIve some examples
*There were two questions based on Linker and Loader . How Compiler converts the HLL to Machine code and where it is stored ,etc
*Difference between IP and MAC , the two address
2- C apti(10 questions)
This section was easy as compared to other section
* two questions were from Pointers .. a 2-D array was declared and various values like (values of arr , of arr[1] , etc were asked )
*A question was asked from Function pointer, a function pointer had function pointer as its arguments . The question asked to identify the arguments and return Type.
*Virtual
*

1- General APTI(10 questions)
*The number 45 is broked into four parts , s.t. The x-2,y-2,z*2 and w/2 accounts to same value find the four words
*Identify the next in sequence:: Z,O,T,T,F,F,S,S,E,N
*In puzzle world one dozen of Pear costs Rs.16, Guava costs Rs. 20 and Grapes costs Rs. 24. How much does Mango costs?
*A car has speed of 72 64 56 in downhill, plain and uphill respectively . A guy travels in the car from Pt. A to pt. B in 4 Hrs and pt. B to pt. A in 4 Hrs and 40 min. what is the distance between A and B?
*A cube 9 27 , hw many coloured 0 ,1,2,3
*A Spider has 8 legs and no wings, Grasshopper has 1 pair of wing and 4 legs  Dragonfly has 2 pair of wings  , 6 legs  . I see 118 legs and  20 pair of wings. How many Spider, grasshopper and Dragon fly are there?
*ABCDE are the digits in order of a 5 digit number and if it is multiplied by 4 it becomes the number EDCBA


Interview Round
*Tell me about yourself
*Tell me about any of your Projects
*Given the string "My name is Sujeet" , list out all possible ways to convert it to "Sujeet is name My".
*Given a link list , give as many ways to reverse it.
*Difference between a Thread and process
*If you have a 32- bit system waht is the maximum possible size of Virtual memory it can can support (later asked Virtual memory only at an instance). what will happen if I put a 6GB RAM in this system
*Implement malloc in C
*Segmentation fault and Stack growing direction question came up during discussion
*Given a network connecting many points , how would u find if it has any disconnected points
*Give any implementation of Thread
*A pattern was given and I asked to loudly think about approaching it
*Why is MAC required other than its security reasons

Search Quality at Yandex

Yandex::The search engine used most in Russia. Here are some of its statistics and information ::

 

Russian Search Market

- Yandex has 60+% market share

- It's all about small attention to details about the search

A Yandex overview

- started in 1997

- no 7 search engine in the world by # of queries

- 150 million queries per day

Variety of Markets

- 15 countries with cyrillic alphabet

- 77 regions in Russia

-> different culture, standard of living, average income, for example: Moscow, Magadan

-> large semi-autonomous ethnic groups (tatar, chech, bashkir)

-> neighbouring bilingual markets

Geo-specific queries

- Relevant result sets very significantly across regions and countries

pFound

- a probablistic measure of user satisfaction

- optimization goal at Yandex sinces 2007

- Similar to ERR, Chapelle 2009 --> hopefully someone can fill in the exact formula

- pFound, pBreak, pRel

Geo-specific Ranking

query -> query + user's region

- may need to build a specific formula for countries/region because of the variance and missing/lacking features in some of them.

Alternatives in Regionalization

- separate local indices or unified indx with geo-coded pages

- one query or region specific query

- query based local intent detection vs. results based local intent detection

- single ranking function vs. co-ranking and re-ranking of local results

- train one formula or train many formulas on local pools

Why use MLR?

Machine learning as a conveyor

- Some query classes require specific ranking

- many features

MatrixNet

A learning method

- boosted decision tree, "oblivious" trees.

- optimize for pFound

- solve regression tasks, train classifiers

Complexity of ranking formulas

20 bytes - 2006

14 kb - 2008

220 kb - 2009

120 MB - 2010

A sequence of More and More complex rankers

- pruning with the static rank (static features)

- use of simply dynamic features (such as bm25)

- complex formula that uses all the features available

- potentially up to million of matrices/trees for the very top documents

- see camazoglu, 2010 early exit optimization

Geo-dependent queries: pFound

- a big jump in 2009 in Quality

- 3x more local results than competitors in Russia, than #2 player

Lessons

- MLR is the only to regional search: it provides us the possiblity of tuning many geo-specific models at the same time.

Challenges

Complexity of the models is increasingly rapidly

-> don't fit into memory!

MLR is in its current setting does not fit well to time-specific queries

-> features of the fresh content are very sparse and temporal

Opacity of results of the MLR

- The backside of ML

Number of featuers grows faster than the number of judgments

-> hard to train ranking

Learning from clicks and user behavior is hard

Tens of GB of data per day!

Yandex and IR

- Participation and Support

- Yandex MLR at IR context

Microsoft Releases Learning to Rank Datasets

Microsoft Research announced that it is releasing a new MS LTR dataset
         After the Recent Learn to Rank Contest held by Yahoo! , Microsoft has presented a similar Dataset :: I am presenting both the dataset's statistics ::

Yahoo! ::. The first dataset had:
29,921 queries
744,692 URLs
519 features

Microsoft :: Released two large scale datasets for research on learning to rank: MSLR-WEB30k with   more than 30,000 queries and a random sampling of it MSLR-WEB10K with 10,000 queries.136 features have been extracted for each query-url pair.The dataset is a retired dataset. What makes this quite interesting is that the features have been released.

You can see the feature list.

See also the Y! LTR datasets.

Lessons and Challenges from Product Search

Lessons and Challenges from Product Search

Daniel Rose, A9

Different Domains, Different Solutions

- Traditional IR,

- Enterprise search

- Web search

- Product Search

How are the issues different? Let's go back to user goals...

The Goals of Web Search

- Understsanding user goals in web search paper

Why do people search on Amazon?

- When they want to buy something?

Even ignoring the non-buying issues..

The Goals of the product Search

- Depends on where you are in the buying funnel.

-- Top: awareness, then Desire, then Interest, finally Action

St. Elmo Lewis, 1898

- Provide the right tools at the right stage in the process.

[roller coaster]

- toys and games

- sort by average customer review

- sort by price (is actually hard: new vs. used, amazon vs. third-party, etc...)

Different Tools for Different Stages

- Product search shows more fluid movement between searching and browsing behavior (relying on faceted metadata)

- Because of the nature of the search task?

- Because of the interfaces?

What Amazon Queries Look Like

- [which old testament book best represent the chronological structure]

- [shipping rates for amazon]

- [long black underbust corset] - still looking

- vs ISBN number -> about to buy it

Context is King

- Some facets for Dresses vs. Digital Cameras

- The problem of facet selection

- Not a one size fits all UI solution for different facet types

- We can interpret your query in a smarter way: [timberland] boots inside shoes is a brand

- Timberland in music -> Timbaland the band

Amazon is a MarketPlace...

- So search must be realtime

-- new products

-- new merchants

-- prices being changed all the time

Structured Data: "It's a gift... and a curse"

- Unlike the web search, we know the semantics of different bits of text

- We know what fields are important for customers (e.g. brand)

- A large degree of quality control (less adversarial problems)

- We don't have to do sentiment analysis to know if a review is positive/negative

A Curse

- Search engine needs to have both DBMS-like "right answer" behavior and IR-like "best answer" behavior

- Tradiontional IR mechanisms don't always work well for structured data

-- e.g. naive tf x idf

What happens when one of the fields is order of magnitudes bigger than others?

-- Search inside the book vs. brand name

- What happens when you don't have all the fields all the time? (missing data)

-- ratings, reviews correlate with user satisfaction, but it may not be there

Using Behavioral Data

- Powerful source of information for any search engine

- When is using behavioural data an invasion of privacy (or just plain creep), and when is it better for users?

- Customers of a business seem more comfortable with that business learning from past behavior.

Interpreting Behavioral Signals

Example: Are search result clicks good and bad?

- How many clicks are best?

-- 1: the customer found what their are looking for right away

-- many: comparison shopping and are looking around at multiple items

-- zero: the search result contained all the information necessary

Also, some items are inherently "click attractive", e.g. a book with a sexy cover

Why is web search so hard... to evaluation? 2004

Evaluating Product Search Relevance

Common argument

-- Customers to to a shopping site to buy stuff

-- if a search engine change leads to customers buying mor stuff, they must have had their search need met more effectively.

-- Therefore, relevance can be measured by how much customers buy.

What's wrong with this argument?

-- besides ignoring the rest of the buying funnel, and that someone is ready to buy.

The A/B Test Mystery

- Compare ranking algorithms A and B

- Assign half of users A and half to B

- And the end the avg. revenue is higher in A than B.

-> algorithm A could be better than B, or Algorithm A could be recommending higher priced items than B

-> Algorithm A could be recommending completely unrelated, but very popular items.

Lessons from IR

One idea: Generalizing the buying funnel

- The information seeking funnel

- Wandering: no information seeking goal in mind

- Exploring: have a general goal, but not a plan on achieving

-Seeking: have started to identify info needs that must be satisfied, but needs are open-ended

-Asking: have a very specific information need corresping to a closed class question

Published in: The information seeking funnel, 2008.

Summary

- Start thinking about how to meet user needs before user knows she has a need

- Offer different interaction mechanisms for different parts of the information seeking process

- Let type of content influence the way search works

- Design for realtime

- Interpret behavioral data carefully

- Exploit Structure when have it

- Exploit context when you have it

I don't have permission to install a Perl module on the system!

     I was trying to install a perl module lately and I got through this error. I was trying to install Math:Combinatorials

robern@robern-laptop:/media/EE6A5C156A5BD8C3/Users/ROBERN/Desktop/iiit/LTRC_work/math_combinatory_perl/Math-Combinatorics-0.09$ make install
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ERROR: Can't create '/usr/local/man/man3'
mkdir /usr/local/man/man3: Permission denied at /usr/share/perl/5.10/ExtUtils/Install.pm line 479

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 at -e line 1
make: *** [pure_site_install] Error 13

If you don't have root permission you will not be able to install a module in the usual place on a shared user system. If you do not have root access you may get errors like:

 This is easy to get around. You just install it locally in your home directory. Make a directory called say /lib in your home directory like this:

# first navigate to your home directory
$ cd ~

# now make a directory called lib
# on UNIX
$ mkdir lib
# on Win32
C:\> md lib

Now you have a directory called ~/lib where the ~ represents the path to your home dir. ~ literally means your home dir but you knew that already. All you need to do is add a modifier to your perl Makefile.PL command

$ perl Makefile.PL PREFIX=~/lib LIB=~/lib

This tell MakeMaker to install the files in the lib directory in your home directory. You then just make/nmake as ::

$ make
$ make test
$ make install

    To use the module you just need to add ~/lib to @INC. See Simple Module Tutorial for full details of how. In a nutshell the top of your scripts will look like this:

#!/usr/bin/perl -w
use strict;
# add your ~/lib dir to @INC
use lib '/usr/home/your_home_dir/lib/';
# proceed as usual
use Some::Module;

Caffeine : The new Search Indexing system of Google

All people who have worked or read about search engines must be pretty aware of the fact that indexing is a pretty tedious job to be done and plays a central role. At research level we generally use libraries like Beautiful Soup for indexing. But the need for a faster indexing is felt for pages which get constantly updated. Therefore search engines usually identify such pages. Web pages such as news, share markets page, etc need to be constantly updated . So these pages are crawled more oftenly as compared to other pages.
      The introduction of Caffine produces a whole different approach and promises to keep all the web pages updated. A parallel processing approach is taken and hundreds of thousands of pages are crawled every second. This leads to a fresher return of query. "Caffeine takes up nearly 100 million gigabytes of storage in one database and adds new information at a rate of hundreds of thousands of gigabytes per day".

Our old index had several layers, some of which were refreshed at a faster rate than others; the main layer would update every couple of weeks. To refresh a layer of the old index, we would analyze the entire web, which meant there was a significant delay between when we found a page and made it available to you.

With Caffeine, we analyze the web in small portions and update our search index on a continuous basis, globally. As we find new pages, or new information on existing pages, we can add these straight to the index. That means you can find fresher information than ever before—no matter when or where it was published.s.

The image compares the old search indexing and the new one: Caffine

Maltparser: A parser to build parser: Introduction & Installation

While a traditional parser-generator constructs a parser given a grammar, a data-driven parser-generator constructs a parser given a treebank. MaltParser is a system for data-driven dependency parsing, which can be used to induce a parsing model from treebank data and to parse new data using an induced model.
MaltParser implements seven deterministic parsing algorithms:

• Nivre arc-eager
• Nivre arc-standard
• Covington non-projective
• Covington projective
• Stack projective
• Stack swap-eager
• Stack swap-lazy

 You can choose whichever algorithm you intend to apply from command promt using Options(see further)

MaltParser currently includes two machine learning package :

• LIBSVM - A Library for Support Vector Machine.
• LIBLINEAR -- A Library for Large Linear Classification
  
  The latest version of MaltParser is available from the MaltParser download page
  

I discuss installation of maltparser on Ubutu Platform. I guess it works on all other platform , if you face any problem, write to me.
Download the Zip File  : malt 1.3.1 
Check if there are newer versions available.
Download the zip file to the directory where you want to extract it or copy it to the directory.
open your command prompt and set the path to the directory . Now type

prompt> unzip malt-1.3.1.zip

To run MaltParser type the following at the command line prompt (it is important that you are in the malt-1.3.1 directory):

prompt> java -jar malt.jar
If MaltParser displays something like the message below, the distribution has been installed successfully.

-----------------------------------------------------------------------------
                          MaltParser 1.3.1
-----------------------------------------------------------------------------
         MALT (Models and Algorithms for Language Technology) Group
             Vaxjo University and Uppsala University
                             Sweden
-----------------------------------------------------------------------------

Usage:
   java -jar malt.jar -f  
   java -jar malt.jar -h for more help and options

help                  (  -h) : Show options
-----------------------------------------------------------------------------
option_file           (  -f) : Path to option file
-----------------------------------------------------------------------------
verbosity            *(  -v) : Verbosity level
  debug      - Logging of debugging messages
  error      - Logging of error events
  fatal      - Logging of very severe error events
  info       - Logging of informational messages
  off        - Logging turned off
  warn       - Logging of harmful situations
-----------------------------------------------------------------------------

Documentation: docs/index.html

To be able to build MaltParser you need Java SE Development Kit (JDK) 6 and Apache Ant 1.7.0. If you dont have these tools please download and install them by following the instructions on their web sites.

To build the MaltParser distribution:

prompt> cd 
prompt> ant dist

should be replaced by the actual path to the directory where malt-1.3.1 is installed.

Morphological Parsing

Parsing in general means, taking an input and producing some sort of structure for it. In NLP the structure could be in various forms like:
1)Morphological
2)Syntactic
3)Semantic
4)Pragmatic

Morphological Parsing takes as input the inflected "surface form" of each word in a text. As output, it produces the parsed form consisting of a canonical form (or lemma) of the word and a set of "tag" showing its syntactical category and morphological characteristics.

Morphological generation is the inverse of morphological analysis, namely the process of converting the internal representation of a word to its surface form. For example, if our internal representations of the words mice and move are:

Root: MOUSE  PartOfSpeech:Noun   Number:Plural
Root: MOVE   PartOfSpeech:Verb   Tense:Past

then morphological generation would convert these to the character strings mice and moved


The morphological parser gets its information from the extraction of:
1)Lexicon: It lists stem and affixes together with basic information about them
2)Morphaotactics: It describes way morphemes are arranged or touch each other
3)Orthographic Rules: These are spelling rules that specify changes that occur when the ordering when two given morphemes combine.

IIIT-Hyderabad And NLP

I am at the International Institute of Information Technology, Hyderabad and working on NLP for LTRC(Language Technologies Research Center)
ICON(International Conference on Natural Language Processing) conference series is a forum for promoting interaction among researchers in the field of Natural Language Processing in India and abroad. Various interesting papers, have been presented lately in the conference.
The SIEL(Search and Information Extraction Lab) has various interesting project. One which I found very interesting was cloud computing course. The lab is collaborating with Yahoo!, which are the biggest users and promoters of Apache Hadoop. SIEL has developed a very close research partnership with Yahoo! R&D Bangalore.

New links and materials

OpenLibrary Relaunch- A redesigned OpenLibrary website was launched last week, designed by Caterina Flake. The new site is like Wikipedia meets a Library catalog. See their blog for details.

LikeLike - An implementation of LSH written for Hadoop. There isn't much documentation on how it is parallelizing the computation.

Inferring Web Searcher Intent Tutorial - The slides from Eugene Agichtein's WWW 2010 tutorial are now available. The first part of the tutorial provides an overview of user task and behaviour models. The second part focuses on utilizing implicit feedback from clicks and other interaction activity.

Yahoo! Learning to Rank Challenge at ICML 2010

Yahoo! has announced a Learning to Rank challenge as part of the Learning to Rank Workshop at ICML 2010.

They are releasing (to participants) two large real-world datasets. The first dataset has:
29,921 queries
744,692 URLs
519 features

The second datasets consists of
6,330 queries
172870 URLs
596 features
There are total 700 different features

There are two tracks

1) Standard LTR track
2)Transfer-learning track
The queries, urls and features descriptions are not disclosed, only the feature values.

Learning to Rank task usually requires large datasets, the more data better chances of making a decision. That is one of the main reasons of Google's success as it gets large hits. The ability to account for all features could be a challenge. The submissions are already on and more than 150 submissions have been made. Multiple submissions are possible.
The format for each of the files is the same as the one used in SVMLight

Relevant papers could be found at:
1)ICML has provided link to 100 papers although I am not able to access them yet
2)CIKM 2008 papers and its lectures
3)AIR WEB papers

Terrier 3.0

Terrier 3.0 has been just released. It is a open source search engine. Its great features are that it is highly flexible and TREC testing for datasets could be easily performed in its default format. A great resource for researchers to experiment.
The search engine is maintained by the IR group at the University of Glasgow . The list of current Project's topics are quite tempting.
Terrier has extensive documentation about indexing and retrieval, as well as about running experiments using TREC test collections.

Mail Anti-spamming

We have all been annoyed with spam mails. The need for spam removal is critically important otherwise we would spend hours daily to remove them from our mailbox.
A mail can be said as spam based on two criteria:
1)Bulk: the recipient's personal identity and context are irrelevant because the message is equally applicable to many other potential recipients.
2)Unsolicited: the recipient has not verifiably granted deliberate, explicit, and still-revocable permission for it to be sent.
The main goal of any anti-spam is that none of the mails which are not anti-spam should be classified as spam no matter how powerful and effective it is.An ideal anti-spam system rejects messages which are both bulk and unsolicited, letting pass those messages which are of specific personal relevance to the recipient (not "bulk"), and those which the recipient has expressly requested (not "unsolicited").
The efficiency plays a major role is mail anti-spamming. The issue of computation for classifying the mail could be a major concern. The parsing of every text message and computation and application of any comparisons, algorithms or statistical analysis could for sure take a lot of time and processing. A system that accepts all mail and then discards the portion which is spam wastes significant resources on mail that will ultimately be discarded. This is the hidden cost of spam, and it can be arbitrarily large, since it depends on how much spam other parties send to the recipient. To address this, the hypothetical intelligent agent could operate at the sender's system, preventing unwanted data from entering the network at all. Unfortunately this seems practically untenable for several obvious reasons, not the least of which is the cost of replicating the agent at every prospective sender. The text analysis based on statistical data is most common approach. It tries to identify the unique trades in spam mails and uses a combination of them to identify spam mails. Using a slightly tweaked (as described below) Bayesian filter, we now miss less than 5 per 1000 spams, with 0 false positives. Characteristics like starting with “Dear friend” or a click invitation can be some of many characteristics to be identified .
Some other types of mail anti-spam are:
The most effective mean is “Source Address Blacklisting” . It is an aggressive approach which refuses all mail from sources which have a known bad history of sending spam, a bad reputation for the same, or some other feature which warrants blacklisting as a bad risk. There are also other applications for general lists of IP addresses, but refusing delivery of mail before "DATA" in SMTP . Unlike most anti-spam techniques, blacklisting reduces the hidden cost of spam by preventing transmission of the message.
Whitelisting is effective as an anti-spam technique, but it is overkill. It eliminates all sources which are not pre-approved, and so long as all the approved sources can be trusted to operate within the bounds of acceptable behaviour, it eliminates spam. It also eliminates any possibility of using the email address in question as a means of introduction
Greylisting eliminates those senders which attempt delivery in a "hit and run" manner, not reattempting delivery in accordance with standards. This has nothing to do with the characteristics of spam in a direct sense, but it so happens that many spammers use "ratware" delivery systems which are egregiously non-compliant with regards to standards, and this technique efficiently prevents communication of messages from such systems.
Challenge/response, or ‘move to spam’ in its broadest sense, attempts to determine that some source address of the message is monitored by a human being capable of taking some requested action. This effectively precludes the possibility that the message is sent in bulk, in most cases.